Create script to generate Flatpak manifest from PIP packages

tingping (50)in #utopian-io • 7 years ago (edited)

The tool to generate flatpaks, flatpak-builder is designed to be sandboxed and generate reproducible results; The downside to this is that some existing tooling such as npm or pip don't necessarily map well to this as they by default download dependencies and packages generally have a lot of dependencies which you wouldn't want to list by hand.

So the solution to this is simply to have automated tooling to generate a json file containing all of the dependencies and include them in your manifest.

The information we need to do this is the list of packages, their sha256, and a URL to download them. Thankfully pip has a download option which resolves the list of all dependencies and downloads them for us so we can get the hash ourselves.

To get the download URL we have to use pypi's API at https://pypi.python.org/pypi/{package_name}/json and find the releases url that matches the filename pip grabbed earlier which is easy enough.

Now that flatpak-builder can reliably download static versions of dependencies we need to install it. The pip install tool does allow looking up packages in a local directory so we can pass it --no-index --find-links "file://${PWD}".

This solution did not turn out perfectly though. For a reason I've yet to track down some packages, for example mypy, have actual dependencies that are not listed or pulled in which is frustrating and currently just worked around by allowing you to pass multiple packages at once so you can manually specify anything missing which seems acceptable for now.

The end result looks like this:

{
    "name": "jedi",
    "buildsystem": "simple",
    "build-commands": [
        "pip3 install --no-index --find-links \"file://${PWD}\" --prefix=/app jedi"
    ],
    "sources": [
        {
            "type": "file",
            "url": "https://pypi.python.org/packages/14/3b/15cfd9c7a8bd9e3a2fe956e20fbc4e7c5768e06aea347d8eb68a05a71653/parso-0.1.1.tar.gz",
            "sha256": "5815f3fe254e5665f3c5d6f54f086c2502035cb631a91341591b5a564203cffb"
        },
        {
            "type": "file",
            "url": "https://pypi.python.org/packages/d2/41/430b325e411d564b1afc37bc7286c3549f4c415ada750a617fb1943c593d/jedi-0.11.1.tar.gz",
            "sha256": "d6e799d04d1ade9459ed0f20de47c32f2285438956a677d083d3c98def59fa97"
        }
    ]
}

Merge request: https://github.com/flatpak/flatpak-builder-tools/pull/6

Posted on Utopian.io - Rewarding Open Source Contributors

#flatpak #python #linux

7 years ago in #utopian-io by tingping (50)

$98.68

12 votes

Sort:

Trending

[-]

dutch (61) 7 years ago

Thank you for the contribution. It has been approved.

You can contact us on Discord.
[utopian-moderator]

$0.00

[-]

tingping (50) 7 years ago (edited)

Do you know why the payout of this post is so much lower than the others?

EDIT: Nevermind it was just delayed.

$0.00

[-]

utopian-io (71) 7 years ago

Hey @tingping I am @utopian-io. I have just upvoted you!

Achievements

You have less than 500 followers. Just gave you a gift to help you succeed!
Seems like you contribute quite often. AMAZING!

Community-Driven Witness!

I am the first and only Steem Community-Driven Witness. Participate on Discord. Lets GROW TOGETHER!

Vote for my Witness With SteemConnect
Proxy vote to Utopian Witness with SteemConnect
Or vote/proxy on Steemit Witnesses

Up-vote this comment to grow my power and help Open Source contributions like this one. Want to chat? Join me on Discord https://discord.gg/Pc8HG9x

$0.00

[-]

steemitboard (66) 6 years ago

Congratulations @tingping! You received a personal award!