Python: main

Breaking down the Python from https://github.com/jorgebastida/awslogs

It’s installed with:

 

Files are packaged using a setup.py file. E.g.

https://github.com/jorgebastida/awslogs/blob/master/setup.py

https://packaging.python.org/tutorials/packaging-projects/

 

Here’s a sample script (/usr/local/bin/awslogs – source code: https://github.com/jorgebastida/awslogs):

 

#!/usr/local/opt/python@2/bin/python2.7

Path to Python. No idea why there’s an @.

https://stackoverflow.com/questions/53482939/why-is-there-an-symbol-in-the-shebang-of-my-python-script

 

# -*- coding: utf-8 -*-

Specifies the encoding.

https://stackoverflow.com/questions/4872007/where-does-this-come-from-coding-utf-8

 

Imported as used later on. E.g. sys.argv[0]

 

from awslogs.bin import main

means: from the package awslogs.bin import the module main.

https://docs.python.org/2/tutorial/modules.html#more-on-modules

How does Python find these packages?

They’re in it’s path. E.g.

and

What is this dist-info directory?

It’s created by Wheel – one of Python’s packaging formats. See: https://packaging.python.org/discussions/wheel-vs-egg/

and

the mainfunction is from awslogs/bin.py here:

/usr/local/lib/python2.7/site-packages/awslogs/bin.py

More on sys.path: https://leemendelowitz.github.io/blog/how-does-python-find-packages.html

 

if __name__ == '__main__':

If run from the command line (e.g. python this-file.py ) then __name__ will == '__main__'. Otherwise, if it’s been run as a module then __name__ will be the name of the module.

https://stackoverflow.com/questions/419163/what-does-if-name-main-do

 

sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])

https://stackoverflow.com/questions/53483812/what-does-this-line-do-sys-argv0-re-subr-script-pyw-exe-s

https://www.pythonforbeginners.com/system/python-sys-argv

 

sys.exit(main())

https://stackoverflow.com/questions/5280203/what-does-this-mean-exit-main

 

Python > parsing JSON

Given:

Note: [] are for lists, {} are for dictionaries.

We can parse it with Python like this:

 

https://stackoverflow.com/questions/2835559/parsing-values-from-a-json-file

 

Python: getting rid of the ‘u’ symbol when outputting text

I find it really annoying that whenever Python prints a string it prints out a u when it’s Unicode.

When you copy and paste that stuff it retains it all. If you’re copying and pasting JSON you’re left with broken JSON. Here’s how to get rid of it:

 

Python > Inspect an object

Use

print dir(<my object>)

where <my object> is your object name.

https://stackoverflow.com/questions/192109/is-there-a-built-in-function-to-print-all-the-current-properties-and-values-of-a?noredirect=1&lq=1

 

dirreturns the list of names in the current local scope.

https://docs.python.org/3/library/functions.html#dir

 

vars returns the __dict__attribute for a module, class or instance.

https://docs.python.org/3/library/functions.html#vars

 

__dict__ is a dictionary or other mapping object used to store an object’s writable attributes.

https://docs.python.org/3/library/stdtypes.html#object.__dict__

 

Isolated Development Environments with virtualenv

Install

pip install virtualenv

Create a directory for your virtualenvs

mkdir .virtualenvs
cd .virtualenvs

Use virtualenv

Let’s say we want to create a virtualenv called test:

virtualenv test

. test/bin/activate

and the prompt should change to show:

(test)

Now, installing a package will install it in this virtualenv. E.g.  pip install pylintwill install pylintinto this virtualenv under .virtualenvs/test/bin

and the PATH is changed so you use stuff under this virtualenv. E.g. which python:

.virtualenvs/test/bin/python

Deactivate virtualenv (also clears prompt with brackets): deactivate

Activate virtualenv: . ./test/bin/activate

virtualenvwrapper

Install:

pip install virtualenvwrapper

Edit ~/.profile

source /usr/local/bin/virtualenvwrapper.sh

export PROJECT_HOME="$HOME/Code/Python"

This gives us some extra commands – mkvirtualenv, setvirtualenvproject, workon:

mkvirtualenv <virtualenv>

Makes the virtualenv in your .virtualenvs directory.

 

workon <virtualenv>

E.g. workon test

This doesn’t change us to the code directory though. You can do this by cding to the code directory and using:

setvirtualenvproject

This ties the current directory with the virtualenv. Handy if you need to move directories.

 

mkproject – creates a project in your code directory, sets up a virtualenv, cd’s to the code directory and activates the virtualenv.

http://virtualenvwrapper.readthedocs.io/en/latest/command_ref.html

 

Python Packages – use setuptools and pip

Many ways of creating, distributing and installing packages. E.g.

distutils, setuptools, easy_install, ez_setup, pip.

Multiple formats:

egg, wheel, source dist.

To clear up this mess, the Python Packaging Authorits (PyPA) was setup. Advice is, if creating and distributing a package:

  1. use setuptools

Note: distutils is discontinued and merged into setuptools. easy_install is also merged into setuptools.

If installing a package, use pip and virtualenv.

Note: from Python 3.4, pip comes pre-installed.

Also homebrew installs pip by default with Python.

 

Notes on  pip

pip list

pip show <specific package>

pip search query

Requirements file

pip freeze > requirements.txt: save project dependencies into requirements.txt

pip install -r requirements.txt: install packages listed in file

 

Why Python sucks – why I hate Python: a Python rant

Suck #1

I started following a super-basic tutorial on Python as I’d learned the basics over the years and wanted to learn again from scratch.

The very first thing they do is:

python

>> import requests

Simple, right?

No!

I got:

So now I’m spending half an hour trying to debug the most basic thing in a Python tutorial. This is clearly a sign of things to come.

Various StackOverflow answers describing complicated solutions that did not help:

https://stackoverflow.com/questions/38836249/error-when-import-requests-no-module-named-requests

https://stackoverflow.com/questions/46751905/import-error-importerror-no-module-named-requests

Turns out the solution was to do:

pip install requests 

https://stackoverflow.com/questions/17309288/importerror-no-module-named-requests

Fortunately, I had pip installed. But basically a complete beginner would have that as an additional hurdle to overcome.

Note:

''=> current working directory

then searches through directories.

/usr/local/lib/python2.7/site-packages => third party packages

Note: on Debian this is called dist-packages.

Suck #2

Following a script that mentions os.environ I get:

NameError: global name 'os' is not defined

Turns out I need to import os. Python folks – can’t this be done automatically? It’s not hard. Just see a variable that matches a package you know and import it. DONE. Kind of like the Mac’s AutoSave. Let’s get out of the ’80s here…

https://stackoverflow.com/questions/23048129/python-name-os-is-not-defined

Suck #3

Again, following that script I copy and paste part into a REPL to try it out. As the indentation is different I get:

IndentationError: unexpected indent

Groan. So, I have to go through the script manually adding indents. Again, let’s get rid of indent restrictions. Stupid.

Suck #4

Gibberish logs and debug output.

A method I had copied was clearly wrong. When I attempted to use it from the REPL with ec2 = get_client("ec2") I got:

What a pile of gibberish. To be clear – 21 lines and 1629 characters of gibberish.

Why not print it out in Assembly just to make it less clear?! Has anyone made any effort whatsoever on making this more user-friendly?

Suck #5

I’ve got an object that’s been returned using a pretty standard library. i.e.

client = boto3.client('ec2')

There is no output when I run it from the REPL. Sounds OK.

Let’s inspect this object. Looking for introspectionI find:

https://stackoverflow.com/questions/1006169/how-do-i-look-inside-a-python-object

Ooh. Python has a strong set of introspection features. OK, let’s try

client.type()

Jeez! Another 6 lines of gibberish.

How about client.getattr()? Surely that’s got to return something?!

Why can’t they just return messages like: AttributeError: 'EC2' object has no attribute 'getattr' and get it over with. Then you could have some debug parameter to find out more if any of the additional text above is actually useful.

At least the very user-friendly underscore underscore dict underscore underscore method (i.e. __dict__) helps figure out what’s going on behind the scenes. That was an easy one to figure out without digging through Stackoverflow. Not.

Suck #6

Let’s follow the 4 simple steps here:

https://github.com/ElasticHQ/elasticsearch-HQ

1. Download or clone the repository.

No problem.

2. Open terminal and point to root of repository. Type: pip install -r requirements.txt

I get (in red):

awscli 1.15.53 has requirement botocore==1.10.52, but you’ll have botocore 1.12.23 which is incompatible.

I’ve no idea now whether I need to upgrade awscli, downgrade botocore. Or even why I’m getting this message.

It also fails with:

Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: ‘/usr/local/man’
Consider using the --user option or check the permissions.

Why on earth didn’t the maintainers bother to mention that you might need to use --user in the install instructions.

I run:

pip install --user -r requirements.txt

and no longer see the red fails. I’ve no idea why the awscli, botocore errors have disappeared.

3. Run server with: python3 application.py. Alternatively: ./manage.py runserver

Great. 2 options to fail.

Let’s try the second:

./manage.py runserver

and the nice simple error message:

I don’t understand why they couldn’t make this less simple to understand. Perhaps another 1500 lines of gibberish to reduce legibility would have helped. E.g. they could have thrown in some random characters to help confuse the end user.

Given that the instructions say:

Requirements
Python 3.4+

and I find this issue buried on github:

https://github.com/ElasticHQ/elasticsearch-HQ/issues/352

which helpfully suggests ignoring the official (faulty) install instructions and using (the also incorrect):

pip3 install -r requirements.txt

The correct install instructions are:

pip3 install --user -r requirements.txt

https://github.com/ElasticHQ/elasticsearch-HQ/issues/352#issuecomment-370993165

Handily there’s also no mention that:

python3 application.py

will take 10 seconds before it starts generating output and so appearing it’s not working.

4. Access HQ with: http://localhost:5000

Finally!

Did I say I hate Python?

 

Suck #7

Why on earth use 4 spaces?!

Let me rephrase it. If you use one less or more of an invisible character your code will break. Or if you swap it out for another invisible character that looks exactly the same (i.e. the tab) it will also break.

Brilliant design idea.

Let’s find something analogous in the real world.

If you reach for the right glass you can have a drink of water.

But if you reach for the wrong glass (that looks exactly the same) it breaks.