TIL: Dynamic variable names in Bash

08 Apr 2016 by Uroš Trebec

From time to time one need to do some Bash scripting. Lately I’ve come back to this while writing a Docker entrypoint for a new service we’re launching.

Docker entrypoint scripts are usualy where you try to mangle together some environment variables into other environment variables to make use of in the application.

Today I learned how to dynamicaly create a variable and then use it to fetch a value from existing (environment) variable.

#!/bin/bash
set -e

# Using this for the dynamic part
COUNT=1

# The VALUE must to be set from an existing variable, but don't know exactly how its named
until [ -n "$VALUE" ]; do
    # Construct a dynamic variable name
    NAME="SOME_DYNAMYCALY_NAMED_VARIABLE_${COUNT}"

    # Let me know what's going on
    >&2 echo "VALUE is not set: trying ${NAME}"

    # Try to se the VALUE by getting the value of the variable named  i.e. SOME_DYNAMYCALY_NAMED_VARIABLE_1.
    # The magic is in the ${!NAME}, which sets the value of $SOME_DYNAMYCALY_NAMED_VARIABLE_1 to the VALUE variable.
    export VALUE=${!NAME}
done

The whole magic (and something new for me) is this:

${!NAME}

Weird Exception While Testing

20 Mar 2016 by Žan Anderle

Tracebacks are crucial part of a coding process. Even more so when writing tests. So when traceback gives you inaccurate information, it can really mess with you. Last week I spent way too long on an exception that should have been trivial. The reason? Incomplete traceback.

I wrote a test for email templates in test_email_templates.py with the following imports:

# -*- coding: utf-8 -*-
from django.test import TestCase
from django.template.loader import get_template, TemplateDoesNotExist
from django.utils.translations import override
from django.conf import settings
from django.contrib.sites.models import Site

After I wrote the test, I ran the test which resulted in a weird exception

$ ./manage.py test my_app.tests.test_email_templates
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/zanderle/.virtualenvs/env/lib/python3.4/site-packages/django/core/management/__init__.py", line 351, in execute_from_command_line
    utility.execute()
  File "/Users/zanderle/.virtualenvs/env/lib/python3.4/site-packages/django/core/management/__init__.py", line 343, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/zanderle/.virtualenvs/env/lib/python3.4/site-packages/django/core/management/commands/test.py", line 30, in run_from_argv
    super(Command, self).run_from_argv(argv)
  File "/Users/zanderle/.virtualenvs/env/lib/python3.4/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/zanderle/.virtualenvs/env/lib/python3.4/site-packages/django/core/management/commands/test.py", line 74, in execute
    super(Command, self).execute(*args, **options)
  File "/Users/zanderle/.virtualenvs/env/lib/python3.4/site-packages/raven/contrib/django/management/__init__.py", line 41, in new_execute
    return original_func(self, *args, **kwargs)
  File "/Users/zanderle/.virtualenvs/env/lib/python3.4/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/Users/zanderle/.virtualenvs/env/lib/python3.4/site-packages/django/core/management/commands/test.py", line 90, in handle
    failures = test_runner.run_tests(test_labels)
  File "/Users/zanderle/.virtualenvs/env/lib/python3.4/site-packages/django/test/runner.py", line 209, in run_tests
    suite = self.build_suite(test_labels, extra_tests)
  File "/Users/zanderle/.virtualenvs/env/lib/python3.4/site-packages/django/test/runner.py", line 121, in build_suite
    tests = self.test_loader.loadTestsFromName(label)
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/loader.py", line 114, in loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_email_templates'

This got me scratching my head and I started looking for possible reasons. I got nowhere for 10 minutes. Eventually I noticed something by chance.

from django.utils.translations import override

There is no translations in django.utils. There is however, django.utils.translation. So simply changing it to

from django.utils.translation import override

Got everything working nicely. I later realized, that I could have figured this out by either running all the tests (or all the tests in the app), or trying to import this specific test. Both of which returns the accurate traceback.

In [1]: import my_app.tests.test_email_templates
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-f8f5ee3652d4> in <module>()
----> 1 import my_app.tests.test_email_templates

/Users/zanderle/Projects/dmt/datafy/my_app/tests/test_email_templates.py in <module>()
      2 from django.test import TestCase
      3 from django.template.loader import get_template, TemplateDoesNotExist
----> 4 from django.utils.translations import override
      5 from django.conf import settings
      6 from django.contrib.sites.models import Site

ImportError: No module named 'django.utils.translations'

When running all the tests in an app:

======================================================================
ERROR: my_app.tests.test_email_templates (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/case.py", line 577, in run
    testMethod()
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/loader.py", line 32, in testFailure
    raise exception
ImportError: Failed to import test module: my_app.tests.test_email_templates
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/loader.py", line 312, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/unittest/loader.py", line 290, in _get_module_from_name
    __import__(name)
  File "/Users/zanderle/Projects/dmt/datafy/my_app/tests/test_email_templates.py", line 4, in <module>
    from django.utils.translations import override
ImportError: No module named 'django.utils.translations'

I still haven’t figured out why this doesn’t show up when running a specific test. Useful to know though.

TIL: docker-compose creates a new container on every run

17 Mar 2016 by Uroš Trebec

We use docker-compose extensively while developing Datafy.it services.

We all know running docker-compose up creates a new container project_service_1. This container is reused when the command is rerun.

Today I learned that each docker-compose run creates new container too. It’s created with the name project_service_run_X, X being the sequence number.

Sooner or later, the container list can looks like this:

$ docker ps -a --filter "name=run" --filter "status=exited"

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                    PORTS               NAMES
1d8f400747f9        excelsior_django    "/entrypoint.sh pytho"   20 hours ago        Exited (0) 20 hours ago                       excelsior_django_run_6
2ecc0fc809ae        excelsior_django    "/entrypoint.sh pytho"   20 hours ago        Exited (1) 20 hours ago                       excelsior_django_run_5
2109c22189cc        db4bad08bc33        "/entrypoint.sh pytho"   20 hours ago        Exited (1) 20 hours ago                       excelsior_django_run_4
ddee814711e4        ff97abc48153        "/entrypoint.sh pytho"   20 hours ago        Exited (1) 20 hours ago                       excelsior_django_run_3
cf51961d0b15        ff97abc48153        "/entrypoint.sh pytho"   20 hours ago        Exited (0) 20 hours ago                       excelsior_django_run_2
590987e41124        ff97abc48153        "/entrypoint.sh pytho"   20 hours ago        Exited (0) 20 hours ago                       excelsior_django_run_1

Fortunately, this mess can be cleaned:

$ docker ps -a -q --filter "name=run" --filter "status=exited" | xargs docker rm

Or avoided in the first place, by using the --rm option:

$ docker-compose run --rm service /bin/bash

TIL: Do Not Store Fixtures In Project Root

11 Mar 2016 by Primož Kerin

Always check for the same name files in your Django project.

If you have two fixtures with the same name, one in your app’s fixtures folder and one in your project dir, it will use the one in your project dir when you run your app’s tests.

So, if you run tests, expecting that your test suite will use my_app/fixtures/companies.json, you’re gonna have a bad time. It will actually use the one in your root directory.

.
├── companies.json
├── my_app
|   ├── fixtures
|   |   └── companies.json