Using Python to Restore JIRA Custom Field Values from Issue History

Consider the following scenario: you move a bunch of issues from one type to another, or from one project to another. You don’t pay much attention and just confirm everything. Only then you find out some custom fields are gone and the data is lost – since the field context does not include the new issue type or project.

Don’t panic! no need to rush to restore the entire instance database from backup! The data is still visible in the issue’s History tab. All you need to do is fix the context of the fields and then restore it from the issue history. In this post I share some Python code (using python-jira module) to help you achieve that.

Step 1: Fix the field’s context

This better be done manually and carefully, if your policy is to maintain strict field configuration (which in general is a good idea since it improves Jira performance).

Go to administration > issues > custom fields and for each “lost” field which you wish to restore, click the cog icon > ‘Configure’ and then ‘Edit Configuration’. Set the proper issue types and projects.

Step 2: Prepare the restoration script

Now let’s prepare a python script to get a single issue key and collects the lost data from the issue history

Step 2a – Connect to JIRA and obtain the issue

# Connect to JIRA
from jira.client import JIRA
jira = JIRA({'server': jira_server_url}, basic_auth=(username, password)

# create mapping for custom field names and types
name_map = dict()
type_map = dict()
for field in jira.fields():
    if field['custom']:
        fname = get_current_field_name(field['name'])
        name_map[fname] = field['id']
        type_map[fname] = field['schema']['type']

# get the issue object - note the 'changelog' setting
issue = jira.issue(issue_key, expand='changelog')

Step 2b – Look for the lost fields in the history and get their original values

# get the values of the lost fields from the history entry where they got nullified
lost_fields = ['field 1', 'field 2']
original_value = dict()
for history in issue.changelog.histories:
    for item in history.items:
        field_name = item.field
        if field_name in lost_fields and not item.toString:
            # get the previous field value - in two forms, see below
            field_value_str = getattr(item, "fromString")  # string value
            field_value = getattr(item, "from")  # raw value (empty for string-based fields)
            field_type = type_map[field_name]
            if field_type == 'datetime':  # need to add milliseconds
                original_value[field_name] = field_value.replace('+', '.000+')
            elif field_type == 'user':  # username
                original_value[field_name] = field_value
            elif field_type == 'group':  # appears as array but we need it as string
                original_value[field_name] = field_value_str.replace('[', '').replace(']', '')
            elif field_type == 'array' and field_value:  # convert to python array - not using "eval" to avoid weird/malicious strings.
                original_value[field_name] = field_value_str.split(',')
            else:
                original_value[field_name] = field_value_str
               

Now debug/test the script on different issues to make sure it can handle the various fields and values you have. If you encounter an exception – most likely the field configuration is not correct, so double-check what you did in step 1.

Step 3 – Add code to restore the collected data

Go over the list of values we collected and update the issue with them. We need to update differently based on the field items, Jira’s REST API is anything but unified… some useful examples can be found here.

Note – we do not update fields which are not empty (e.g. someone re-entered them manually).

fields_to_update = dict()
for field_name in original_value.keys():
    field_real_name = name_map[field_name]
    field_type = type_map[field_name]
    # don't update a field which now has a value
    curr_value = getattr(issue.fields, field_real_name)
    if curr_value is None:
        value = values[field_name]
        if field_type == 'number':
            fields_to_update[field_real_name] = int(value)
        elif field_type in ['string', 'datetime']:
            fields_to_update[field_real_name] = value
        elif field_type in ['user', 'group', 'version']:
            fields_to_update[field_real_name] = {'name': value}
        elif field_type in ['array']:
            # for multi-version pickers, need to use [{'name': x} for x in value] 
            fields_to_update[field_real_name] = [{'value': x} for x in value]
        else:
            fields_to_update[field_real_name] = {'value': value}
        # useful debug print if needed. Helps comparing the outcome against the official REST API.
        # print("Adding {}({},{}) -> {}".format(field_name, field_real_name, field_type, fields_to_update[field_real_name]))
# now update the issue
if fields_to_update:
    try:
        print("Updating issue " + issue.key)
        issue.update(fields=fields_to_update)
    except Exception as e:
        print("{}: FAILED: {}".format(issue.key, e))
else:
    print("{}: Nothing to update".format(issue.key))

That’s it! Run it on the affected issues and everything’s back to normal.

Tip – if you’re going to run this script on a bunch of issues, I highly recommend to disable email notifications in your Jira instance beforehand ūüôā

Quick tip: Using Python’s expandvars() with quoted variables on Windows

I recently encountered an elusive bug, which eventually I tracked down to the fact that environment variables are not replaced in a certain string even though it is set by calling os.path.expandvars(original string).

Turns out, the problem is that expandvars() doesn’t expand variables who are enclosed within a string literal. I couldn’t find it documented anywhere but facts are, this is the case.

Small example:

>>> import os
>>> os.environ['VAR1'] = 'Value1'
>>> s1 = "this is '%VAR1%'"
>>> os.path.expandvars(s1)
"this is '%VAR1%'"
>>> s2 = "this is %VAR1%"
>>> os.path.expandvars(s2)
'this is Value1'

It may be a good idea in some cases, but for me it was quite an annoyance.

So, how to overcome this?

Continue reading

Running Python Unit Test in a VSTS/TFS build

Here’s a short explanation how to execute python unit tests during VSTS/TFS builds, without the need of plug-ins. The basic idea is to create a JUnit XML from your test results, and then have it published to the TFS build dashboard.

First, on the relevant build agent(s), install the pytest module. You can use the following command, or your favorite CM tool.

python -m pip install pytest

Next, edit your build definition in TFS, and add two build steps: one (Batch Script) for running py.test to generate a junit XML, and one (Publish Test Results) for publishing the xml to the test results page.
The full command line to run the tests is as follows (in this example, the unittest code is in test_MyApp.py):

python -m py.test --junitxml test-results.xml test_MyApp.py

Here is a sample build definition which includes only those two steps:

BuildSteps

Build Steps

BatchScript

Batch Script Settings

 

Finally, queue the build, and you will able to see the test results in the build page:

TestResults

 

 

Python and Windows Symbolic Links

Python, like many other popular OSS (Git anyone?), does not support symbolic links on Windows platforms , although they have been around since Windows Vista. Note, I am referring to actual symbolic links, not NTFS directory junctions or shortcuts.

I’m not really sure why is that. Maybe because they do not distinguish between different Windows revisions?

My current approach is¬†to use the OS commands to create/delete symlinks. It’s not very elegant but it works without compatibility issues, unlike other solutions (calling win32 api through DLLs, manipulating file attributes, and other stuff you find in StackOverflow or tech blogs)

For example, to create a symbolic link of directories, one can use:

 child = subprocess.Popen(['MKLINK', '/D', link, target], stdout = subprocess.PIPE, stderr = subprocess.STDOUT, shell = True)
 streamdata = child.communicate()[0]

And check child.returncode for the result (and the output – stdout and stderr combined – available in the streamdata variable)

To remove a symbolic link to a directory, use the windows RMDIR command (os.rmdir or os.unlink won’t work)

 

Python in Visual Studio 2015

When I first heard that Visual Studio 2015 is going to support¬†developing¬†with Python, I wasn’t sure how to react. Microsoft? Python? it just doesn’t seem related. I was very skeptical about working in Python within Visual Studio.

Recently I started a new pet project in GitHub to play around with VS integration to GitHub. So I figured, let’s include¬†some Python code just to¬†see how it feels working with Python in VS.

And it feels great, actually! Visual Studio users would feel right at home with IntelliSense-like auto completion and tooltips, debugging capabilities, unit testing with Test Explorer, advanced searching and editing, and more. I would dare comparing it to some of the best Python IDEs out there like PyCharm. GitHub integration also work seamlessly.

This Python support (a.k.a Python Tools for Visual Studio) is included in the free Visual Studio 2015 Community edition, and even available as open source on GitHub.

Microsoft! on GitHub! Times are changing, indeed.

cmd /c doesn’t like quotes

I was quite baffled at first.

This works:

cmd /c "C:\Program Files\7-Zip\7z.exe" C:\Temp\MyFile.7z

But this fails:

cmd /c "C:\Program Files\7-Zip\7z.exe" "C:\Temp\My Other File.7z"
'C:\Program' is not recognized as an internal or external command, operable program or batch file.

Turns out it’s an issue as old as Windows 2000.

The solution is to add another pair of quotes around the entire command:

cmd /c ""C:\Program Files\7-Zip\7z.exe" "C:\Temp\My Other File.7z""

Thank you, Microsoft.

Getting a JIRA issue’s priority using Perl

JIRA::Client is a great and useful module. Too bad some of its usage is so cryptic!

For example – suppose you want to print an issue’s priority field. if you simply use $issue->{priority} you will get the priority id, not the display name which everyone is used too.
So, how to get the display name? documentation is very vague on this subject. Google doesn’t help much here.

After a lot of trial & error I figured out a way.

Continue reading

ERROR: Value for ‘/TR’ option cannot be more than 261 character(s).

One would expect that, in the year of 2014, such errors would be a thing of the past…
But it’s true. You cannot create/update a scheduled tasks from command-line (using SCHTASKS) if the command line it runs is longer than 261 characters.
The funny thing is that you *can* create such a task using the GUI. So why the command-line limitation?

So, how to work around it?
option 1 – create a batch file to run the full command, and set the schedule tasks to execute that batch file. In most scenarios this is the easiest solution
option 2 – use a 3rd party utility for handling scheduled tasks (there are several, including jt.exe from Microsoft’s Windows 2000 SDK)