Python > parsing JSON

Given:

Note: [] are for lists, {} are for dictionaries.

We can parse it with Python like this:

 

https://stackoverflow.com/questions/2835559/parsing-values-from-a-json-file

 

Python: getting rid of the ‘u’ symbol when outputting text

I find it really annoying that whenever Python prints a string it prints out a u when it’s Unicode.

When you copy and paste that stuff it retains it all. If you’re copying and pasting JSON you’re left with broken JSON. Here’s how to get rid of it:

 

Python > Inspect an object

Use

print dir(<my object>)

where <my object> is your object name.

https://stackoverflow.com/questions/192109/is-there-a-built-in-function-to-print-all-the-current-properties-and-values-of-a?noredirect=1&lq=1

 

dirreturns the list of names in the current local scope.

https://docs.python.org/3/library/functions.html#dir

 

vars returns the __dict__attribute for a module, class or instance.

https://docs.python.org/3/library/functions.html#vars

 

__dict__ is a dictionary or other mapping object used to store an object’s writable attributes.

https://docs.python.org/3/library/stdtypes.html#object.__dict__

 

Isolated Development Environments with virtualenv

Install

pip install virtualenv

Create a directory for your virtualenvs

mkdir .virtualenvs
cd .virtualenvs

Use virtualenv

Let’s say we want to create a virtualenv called test:

virtualenv test

. test/bin/activate

and the prompt should change to show:

(test)

Now, installing a package will install it in this virtualenv. E.g.  pip install pylintwill install pylintinto this virtualenv under .virtualenvs/test/bin

and the PATH is changed so you use stuff under this virtualenv. E.g. which python:

.virtualenvs/test/bin/python

Deactivate virtualenv: deactivate

Activate virtualenv: . ./test/bin/activate

virtualenvwrapper

Install:

pip install virtualenvwrapper

Edit ~/.profile

source /usr/local/bin/virtualenvwrapper.sh

export PROJECT_HOME="$HOME/Code/Python"

This gives us some extra commands – mkvirtualenv, setvirtualenvproject, workon:

mkvirtualenv <virtualenv>

Makes the virtualenv in your .virtualenvs directory.

 

workon <virtualenv>

E.g. workon test

This doesn’t change us to the code directory though. You can do this by cding to the code directory and using:

setvirtualenvproject

This ties the current directory with the virtualenv. Handy if you need to move directories.

 

mkproject – creates a project in your code directory, sets up a virtualenv, cd’s to the code directory and activates the virtualenv.

http://virtualenvwrapper.readthedocs.io/en/latest/command_ref.html

 

Python Packages – use setuptools and pip

Many ways of creating, distributing and installing packages. E.g.

distutils, setuptools, easy_install, ez_setup, pip.

Multiple formats:

egg, wheel, source dist.

To clear up this mess, the Python Packaging Authorits (PyPA) was setup. Advice is, if creating and distributing a package:

  1. use setuptools

Note: distutils is discontinued and merged into setuptools. easy_install is also merged into setuptools.

If installing a package, use pip and virtualenv.

Note: from Python 3.4, pip comes pre-installed.

Also homebrew installs pip by default with Python.

 

Notes on  pip

pip list

pip show <specific package>

pip search query

Requirements file

pip freeze > requirements.txt: save project dependencies into requirements.txt

pip install -r requirements.txt: install packages listed in file

 

Why Python sucks

Suck #1

I started following a super-basic tutorial on Python as I’d learned the basics over the years and wanted to learn again from scratch.

The very first thing they do is:

python

>> import requests

Simple, right?

No!

I got:

So now I’m spending half an hour trying to debug the most basic thing in a Python tutorial. This is clearly a sign of things to come.

Various StackOverflow answers describing complicated solutions that did not help:

https://stackoverflow.com/questions/38836249/error-when-import-requests-no-module-named-requests

https://stackoverflow.com/questions/46751905/import-error-importerror-no-module-named-requests

Turns out the solution was to do:

pip install requests 

https://stackoverflow.com/questions/17309288/importerror-no-module-named-requests

Fortunately, I had pip installed. But basically a complete beginner would have that as an additional hurdle to overcome.

Note:

''=> current working directory

then searches through directories.

/usr/local/lib/python2.7/site-packages => third party packages

Note: on Debian this is called dist-packages.

Suck #2

Following a script that mentions os.environ I get:

NameError: global name 'os' is not defined

Turns out I need to import os. Python folks – can’t this be done automatically? It’s not hard. Just see a variable that matches a package you know and import it. DONE. Kind of like the Mac’s AutoSave. Let’s get out of the ’80s here…

https://stackoverflow.com/questions/23048129/python-name-os-is-not-defined

Suck #3

Again, following that script I copy and paste part into a REPL to try it out. As the indentation is different I get:

IndentationError: unexpected indent

Groan. So, I have to go through the script manually adding indents. Again, let’s get rid of indent restrictions. Stupid.

Suck #4

Gibberish logs and debug output.

A method I had copied was clearly wrong. When I attempted to use it from the REPL with ec2 = get_client("ec2") I got:

What a pile of gibberish. To be clear – 21 lines and 1629 characters of gibberish.

Why not print it out in Assembly just to make it less clear?! Has anyone made any effort whatsoever on making this more user-friendly?

Suck #5

I’ve got an object that’s been returned using a pretty standard library. i.e.

client = boto3.client('ec2')

There is no output when I run it from the REPL. Sounds OK.

Let’s inspect this object. Looking for introspectionI find:

https://stackoverflow.com/questions/1006169/how-do-i-look-inside-a-python-object

Ooh. Python has a strong set of introspection features. OK, let’s try

client.type()

Jeez! Another 6 lines of gibberish.

How about client.getattr()? Surely that’s got to return something?!

Why can’t they just return messages like: AttributeError: 'EC2' object has no attribute 'getattr' and get it over with. Then you could have some debug parameter to find out more if any of the additional text above is actually useful.

At least the very user-friendly underscore underscore dict underscore underscore method (i.e. __dict__) helps figure out what’s going on behind the scenes. That was an easy one to figure out without digging through Stackoverflow. Not.

 

Jinja Templates and JSON with Python

  1. Loading up a Jinja Template and some JSON using Python then outputting that JSON in the template:

k.py

template.j2

api.json

2. Outputting the value of api when done is true

api.json

template.j2

See also:

http://jinja.pocoo.org/docs/dev/templates/#expressions