Does codeql database creation depend on where codeql is invocated? (interpreted languages)

[Naman-ntc]

Relative imports in python and javascript or sys.path amended imports in python will depend on where main.py or javascript code is executed.
How does CodeQL handle these issues when building the code-flow graphs?
And will these cause issues in databases upon data creation from different source-root?

[tausbn]

During extraction, we follow import statements using the machinery that Python provides. This means that changes to sys.path may not be detected, depending on where and how they are applied.

As a rule of thumb, the extractor mostly concerns itself with making sure that all the relevant files have been extracted. Assigning a meaning to the things that have been extracted is mostly done in the CodeQL libraries. For instance, if a Python file starts with a #! /usr/bin/python line, then we treat it as if it is an entry point, and modify things like module names accordingly. In some cases this may result in a module having multiple distinct names, depending on which entry point is used during execution.

As for whether this will cause issues, that really depends on the specific setup.

[Naman-ntc]

I see. So for example in the following data directory structure

main_repo
|
|__ python
    |__main.py (import folder.algo)
    |__folder
        |
        |__algo (import .algo2)
        |__algo2 

From what I understand - CodeQL would extract the python files and even if source root is provided as main_repo instead of python CodeQL will automatically detect python will be the main directory and import stuff nicely?

But what if this is more complicated? Or consider this -- there are independent codes in two folders python and python2 completely disjoint (sharing common pieces of code). Will CodeQL start connecting codes in the two directories?

[tausbn]

From what I understand - CodeQL would extract the python files and even if source root is provided as main_repo instead of python CodeQL will automatically detect python will be the main directory and import stuff nicely?

Hopefully. 🙂 (Though I have not tested the above setup.)

But what if this is more complicated? Or consider this -- there are independent codes in two folders python and python2 completely disjoint (sharing common pieces of code). Will CodeQL start connecting codes in the two directories?

In that case, the analysis may end up connecting the two directories, even though this connection does not exist during runtime. The main problem here is that Python does allow modules to be split up in this way (through what is known as "namespace packages") and so it can be difficult to tell what's really intended.

In general, we suggest avoiding putting multiple unrelated pieces of code in the same repo, especially if they depend on subtly different packages with identical names.

[Naman-ntc]

Hopefully. (Though I have not tested the above setup.)

Where can I get more clarity into this? I am currently analyzing large set of public github repos with CodeQL and wanted to do this precisely.

In that case, the analysis may end up connecting the two directories, even though this connection does not exist during runtime. The main problem here is that Python does allow modules to be split up in this way (through what is known as "namespace packages") and so it can be difficult to tell what's really intended.

Oh, that's very interesting. I assume the same would happen for javascript

In general, we suggest avoiding putting multiple unrelated pieces of code in the same repo, especially if they depend on subtly different packages with identical names.

Thanks, I understand now why two independent folders can be connected. I was doing even worse thing (running codeql on bunch of independent repositories together. Discussion #477 ).

[tausbn]

Where can I get more clarity into this? I am currently analyzing large set of public github repos with CodeQL and wanted to do this precisely.

Ultimately, this boils down to the question of "what is the name of this module?" (which, if answered correctly, will enable one to figure out which imports then refer to this particular module).

The code for determining this can be found in the following file (though beware, it is rather complicated):

https://github.com/github/codeql/blob/main/python/ql/lib/semmle/python/Module.qll

One thing that this code does not handle well right now is when you have modules inside of folders with names that cannot be used as module names (e.g. foo-module which has a -). In this case, the current logic can get confused as it assumes all folders have Python-module-compatible names. Because of this, something in foo-module/foo/bar.py might get confused with something in bar-module/foo/bar.py, since both of those modules appear to have the same name foo.bar.