Our ClearCase repository make heavy use of ClearCase symbolic links. We began an initiative to get rid of them, since they cause lots of problems – from snapshot view size, to tools not supporting them (e.g. WebPack), and Git migration efforts. See also my previous post on this matter.
The first step, of course, is to find all of them. One may think that cleartool find command will provide this functionality, but it’s lackcing. So we came up with this simple command-line:
cleartool ls -l -r <folder> | findstr /C:"symbolic link"
Folder must be a VOB, or a folder inside a VOB. To iterate on all VOBs, you can run:
for /d %i in (<view root>\*) do @cleartool ls -l -r %i | findstr /C:"symbolic link"
Now, we just need to figure out what to do with all of them…
We spent the last year or so building and maintaining a Continuous Integration (CI) system for our major development group. And it was incredibly daunting and frustrating, considering we had to build it around a legacy and problematic version control system like ClearCase Base.
In the textbooks it seems pretty simple: every commit (check-in) to the repository triggers a build, which in turn checks out the latest version of the repository from source control, and run the build process. This way, if a build fail, you know exactly who broke the build – the person who performed the last commit – and if it the build passes, you have full traceability between the build artifacts and the source code they were built from.
Now, let’s try that with ClearCase Base.
ClearCase is a complicated product, and when you encounter performance issues it’s quite difficult to figure out the reason.
IBM has some technotes and articles on the subject, but most of them involve running all sorts of commands and scripts and meticulously analyzing their output (see one example here).
A few months ago it came to my attention that ALMToolBox are working on a web-based, graphical monitoring tool which provides a simple yet powerful overview on system performance, both on the infrastructure level (CPU, I/O etc.) and on the application level (ClearCase objects and commands), and correlate them as seen below.
You can also watch a live demo on this page.
It’s still early in development but it looks promising, and I urge every ClearCase administrator to keep a close eye on this product.
Edit: Even better – it can be used to monitor virtually anything, not just ClearCase servers. In this page you can see a “vitality report” which also includes ClearQuest, JIRA and Bamboo servers.
Visual Annotate is a nice tool which enables you to annotate a file in ClearCase using an easy to use GUI (instead of using the cleartool annotate command line and trying to figure out what the output means).
It provides a clear, color-coded visualization of where each line comes from (which version and which user). There are many features and customization options, and also integration with IDEs and, with the latest version, the ability to integrate with various Issue trackers.
The most useful feature in my eyes is the ability to see where a change really comes from – if a certain line is the result of the merge, You can see (as a tooltip) the source of the merge, helping you understand who actually did the code change, and where. This is a real time-saver.
In their latest release, they also offer a free edition – you can use the tool for free 20 minutes per day. I suggest you go and check it out.
Just a quick heads-up, I’ve been using this tool for years and was surprised to find out many ClearCase administrators are not aware of it. So if you’re one of them, grab it today:
It finally happens…
After using ClearCase for 17 years, we came to a conclusion that we must migrate to a different tool.
The reason: cost. The licensing and support are simply way too expensive compared to other tools on the market. Consider the fact that a single ClearCase license costs about $4500, which is about 6 times the cost other commercial products charge. Roughly the same rate applies on annual support costs. Not to mention the free alternatives such as the well-versed Subversion or the rising star Git.
So… where to start? there are three major aspects to consider:
1. Which Version Control tool to choose?
2. What is the best data migration strategy?
3. What is the best approach for switching the developers and the development environment to the new tool (builds, utilities etc.)?
We will probably have to hire some expert consultant for this project. But I still need to do my own research, so I’d at least know what questions to ask when the time comes.
Yesterday I attended the Israeli launch of PlasticSCM. This is a Source Control Management tool which claims to provide the best branching as merging abilities, with rich GUI and CLI.
I must say that the demonstration was quite impressive, and I’m really intrigued to try the tool myself. They offer a free ‘small team’ edition (which seems to be a common practice these days, and a great one!).
Another thing I really liked is that they don’t have their own propriety database, instead they support a variety of free and commercial RDBMS software. Any ClearCase administrator could tell horror stories about weird data corruptions or backups gone bad…
Pablo, the co-founder of the company, was passionate about making developers realize that source control is useful, not a liability. He promotes the ‘branch per task’ methodology, which is an excellent approach to development – but difficult to effectively implement in most tools.
It is quite obvious that the tool’s developers were heavily influenced by ClearCase, I guess they’re initial motive was to build a “ClearCase done right”. But PlasticSCM is much more than that today.
While I can’t recommend a tool I haven’t tried myself, I definitely think that it should be at least considered a candidate for anyone looking for Version Control tool, either for new developments, or as replacement for existing tool.
During a recent VOB restore I performed, I noticed that the VOB storage pool (the sdft subfolder) contains several large files, prefixed with tmp_ (for example, myvob.vbs/s/sdft/10/1d/tmp_3996.1)
Turns out these are called ‘unreferenced containers’, or ‘debris’.
This handy little Perl script removes all traces of a view which is irremovable via conventional means – e.g. when the view storage is missing or inaccessible.
The script is design to run on UNIX in an interop environment, but can be easily be adjusted to other cases.
## Usage: perl rmview_by_tag.pl <region> <view tag>
$region = $ARGV;
$region || die "Region must be specified!";
$tag = $ARGV;
$tag || die "View tag must be specified!";
## Detect view UUID
@uuid = `cleartool lsview -l -region $region $tag`;
if (/View uuid/)
s/View uuid: //;
$uuid = $_;
# Remove the tag from the registry
system "cleartool rmtag -view -region $region $tag";
# Unregister the view
system "cleartool unreg -view -uuid $uuid";
# Remove view-related records from all VOBs
system "cleartool rmview -all -uuid $uuid";
Reference: Removing a view
A few months ago we migrated to a new VOB server (we have a Solaris/Windows inter-op, Samba-based environment). The transition went smoothly and all seemed well.
Suddenly, after a month or so, some of our users started having problems accessing some files. It all seemed very random, and it was slowly spreading.
In the view log we saw numerous cleartext-related errors:
2012-05-01T12:37:15+03:00 view_server(3124): Error: view_server.exe(3124): Error: Unable to construct cleartext for object "0x96E3D" in VOB "atlas:/disk02/algotec_vobs/3rdparty.vbs
": error detected by ClearCase subsystem
2012-05-01T12:37:15+03:00 view_server(3124): Error: view_server.exe(3124): Error: Type manager "z_whole_copy" failed construct_version operation.
2012-05-01T12:37:15+03:00 view_server(3124): Warning: z_whole_copy: Error: Can't open input file "\\atlas\disk02\algotec_vobs\3rdparty.vbs\s/sdft\e/28/3-6c7f92d47de9400a948cc3a7b52
3a9f9-fb" - Invalid argument
In the samba log we saw the following errors:
[2012/05/01 15:29:15, 1] libads/kerberos_verify.c:442
ads_verify_ticket: krb5_init_context failed (Too many open files)
[2012/05/01 15:29:15, 1] smbd/sesssetup.c:342(reply_spnego_kerberos)
Failed to verify incoming ticket with error NT_STATUS_LOGON_FAILURE!
Since I couldn’t find anything useful in IBM Rational Support website, or via Google, I contacted IBM Technical Support. After several back-and-forth emails, we nailed the problem. Turns out it wasn’t server-side at all!
The affected users were all using a relatively new PC, all installed from the same image. There were two things wrong with this image:
- The CLEARCASE_PRIMARY_GROUP environment variable was not defined.
- The ‘maximum mnodes’ value in the MVFS Performance configuration was set to 800 (suitable for 64-bit Samba) instead of 200 (suitable for 32-bit Samba, which we use).
After fixing this configuration, the problem was gone and everyone was happy again. Quite a challenge, it was…
Helpful link: Determine if a UNIX or Linux application is 32-bit or 64-bit