Miro Hrončok2023-09-04T13:28:33+00:00http://eng.hroncok.czMiro Hrončokmiro@hroncok.czIPython-like RPM Lua interactive console2020-05-14T00:00:00+00:00http://eng.hroncok.cz/2020/05/14/ilua-rpm-console
<p>Recently, I’ve found myself in a position of writing and debugging some Lua based RPM macros and generators:</p>
<ul>
<li><a href="https://github.com/rpm-software-management/rpm/pull/1153">speeding up python(abi) dependency generator by rewriting it from Bash to parametric Lua macro</a></li>
<li><a href="https://src.fedoraproject.org/rpms/python-rpm-generators/pull-request/7">adding automated provides to replace most of the manual %python_provide calls</a></li>
<li><a href="https://src.fedoraproject.org/rpms/python-rpm-macros/pull-request/52">reimplementing %python_provide from scratch as %py_provides</a></li>
</ul>
<p>The capabilities of the <a href="https://rpm.org/user_doc/lua.html">embedded Lua interpreter in RPM</a> are endless. My Lua skills are not. I need to test the code in small chunks to understand what am I doing.</p>
<p><img src="https://i.kym-cdn.com/entries/icons/original/000/008/342/ihave.jpg" alt="meme" /></p>
<p>When I need to do this in Python, I use the <a href="https://ipython.org/">IPython</a> console (my second favorite software in the Python ecosystem after <a href="https://docs.pytest.org/">pytest</a>).</p>
<p><img src="/assets/2020-05-14-ilua-rpm-console/ipython.png" alt="IPython console" /></p>
<p>For Lua, I’ve used the <code class="language-plaintext highlighter-rouge">lua</code> command for a while. It lets me test basic Lua concepts in an interactive console, but it’s not that powerful as the IPython console and it lacks the RPM provided Lua libraries.</p>
<p><img src="/assets/2020-05-14-ilua-rpm-console/lua.png" alt="Basic Lua console" /></p>
<p>I’ve asked Panu (my colleague and an RPM developer I work with when submitting changes to the RPM project) whether there is an interactive console for the Lua interpreter embedded in RPM <a href="https://github.com/rpm-software-management/rpm/pull/1153#discussion_r401514078">and the answer was yes</a>. I can use <code class="language-plaintext highlighter-rouge">rpm --eval "%{lua:rpm.interactive()}"</code>. But it has some quirks.</p>
<p><img src="/assets/2020-05-14-ilua-rpm-console/rpm-interactive.png" alt="Basic RPM Lua interactive shell" /></p>
<p>Mostly, there is no command history or even line editing, and <a href="https://github.com/rpm-software-management/rpm/issues/1215">the output is hard to get</a>.</p>
<h2 id="ilua">ILua</h2>
<p>Naturally, I’ve searched for an IPython like Lua console and I’ve found the <a href="https://github.com/jupyter/jupyter/wiki/Jupyter-kernels">list of Jupyter Kerneles</a> (IPython console is a frontend to <a href="https://jupyter.org/">Jupyter</a>). There are 3 Lua kernels there:</p>
<ul>
<li><a href="https://github.com/neomantra/lua_ipython_kernel">Lua Kernel</a> (discontinued)</li>
<li><a href="https://github.com/pakozm/IPyLua">IPyLua</a> (a fork of the above, not much active either)</li>
<li><a href="https://github.com/guysv/ilua">ILua</a></li>
</ul>
<p>ILua intrigued me mostly because it says right away: <em>“Lua-implementation agnostic, should work with any Lua interpreter out of the box.”</em> That’s exactly what I need. Maybe I can use it with <code class="language-plaintext highlighter-rouge">rpm --eval "%{lua:rpm.interactive()}"</code>.</p>
<p>Turns out I can, <a href="https://github.com/guysv/ilua/issues/10">but it’s not that simple</a>. The used Lua interpreter needs to respect the <code class="language-plaintext highlighter-rouge">$LUA_PATH</code> environment variable and execute the file given to it as a command-line argument. Naturally, the simplistic <code class="language-plaintext highlighter-rouge">rpm --eval "%{lua:rpm.interactive()}"</code> does neither.</p>
<p>So I’ve created a wrapper:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/usr/bin/bash</span>
<span class="nb">exec </span>rpm <span class="nt">--eval</span> <span class="s1">'%{lua:package.path = "'</span><span class="k">${</span><span class="nv">LUA_PATH</span><span class="k">}</span><span class="s1">';" .. package.path;'</span><span class="s2">"</span><span class="si">$(</span><span class="nb">cat</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span><span class="si">)</span><span class="s2">"</span><span class="s1">';rpm.interactive()}'</span>
</code></pre></div></div>
<p>And it mostly worked. Except for the <a href="https://github.com/rpm-software-management/rpm/issues/1215">slight problem with missing print output</a>. (The executed ILua script does the interactivity, so I’ve removed the <code class="language-plaintext highlighter-rouge">rpm.interactive()</code> call at the end.)</p>
<p><img src="/assets/2020-05-14-ilua-rpm-console/bash-wrapper.png" alt="ILua using a Bash wrapper over RPM Lua" /></p>
<p>I’ve tried to figure out how to launch the RPM Lua interpreter, not in the RPM macro expansion mode (that thing eats all the print output until the end). I’ve done a little RPM symbols lookup and source reading and figured out there is an <code class="language-plaintext highlighter-rouge">rpmluaRunScript()</code> function in <code class="language-plaintext highlighter-rouge">librpmio</code>. So I’ve tried to use it:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#!/usr/bin/python3
</span><span class="kn">from</span> <span class="nn">ctypes</span> <span class="kn">import</span> <span class="n">cdll</span><span class="p">,</span> <span class="n">c_char_p</span>
<span class="n">librpmio</span> <span class="o">=</span> <span class="n">cdll</span><span class="p">.</span><span class="n">LoadLibrary</span><span class="p">(</span><span class="s">"librpmio.so.9"</span><span class="p">)</span>
<span class="n">librpmio</span><span class="p">.</span><span class="n">rpmluaRunScript</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="n">c_char_p</span><span class="p">(</span><span class="sa">b</span><span class="s">"rpm.interactive()"</span><span class="p">),</span> <span class="bp">None</span><span class="p">)</span>
</code></pre></div></div>
<p>It works. A little tweaking to support ILua as well as other use cases:</p>
<script src="https://gist.github.com/63c36381e2daa12446cb70c1a30bef2e.js"> </script>
<p>Note by Panu: <em><code class="language-plaintext highlighter-rouge">rpmluaRunScript()</code> and <code class="language-plaintext highlighter-rouge">rpmluaRunScriptFile()</code> are not considered public API and are not available in the public headers on C side, although the symbols are accessible in the ABI. So they are subject to change without further notice, although the likelihood of that happening doesn’t seem that great, they’ve been exactly the way are since their inception 16 years ago.</em></p>
<p>To use this, put the script on your <code class="language-plaintext highlighter-rouge">$PATH</code> and invoke <code class="language-plaintext highlighter-rouge">ilua -i <name_of_the_script></code>.</p>
<p>And now I have an interactive IPython-like shell for RPM embedded Lua with command history, line editing, completion and more:</p>
<p><img src="/assets/2020-05-14-ilua-rpm-console/irpmlua.png" alt="ILua using a Python ctypes wrapper over RPM Lua" /></p>
<p>Enjoy! PS: ILua is on <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1834280">package review for Fedora</a>, but can be safely pip-installed in the meantime.</p>
<hr />
<h2 id="update-from-2022">Update from 2022</h2>
<p>On RPM 4.18, the script above does not work, but you can use this instead:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/usr/bin/sh -eu</span>
/usr/bin/rpmlua <span class="nt">-e</span> <span class="s1">'package.path = os.getenv("LUA_PATH") .. ";" .. package.path'</span> <span class="s2">"</span><span class="nv">$@</span><span class="s2">"</span>
</code></pre></div></div>
PyCon CZ 2015 and Fedora booth2016-01-18T00:00:00+00:00http://eng.hroncok.cz/2016/01/18/pycon-cz-2015
<p>In November, we’ve been to <a href="https://cz.pycon.org/2015/">PyCon CZ 2015</a>. And when I say <em>we</em> I mean Fedora Python maint and friends team: Matěj Stuchlík, Robert Kuska, Slávek Kabrda, Michal Cyprian, Michal Srb and me.</p>
<p>PyCon CZ 2015 was the first PyCon in the Czech Republic. With Fedora being very Python friendly and with our “Python 3 as Default” feature, we thought we had to be there to represent Fedora. We actually did that in 2 ways:</p>
<h2 id="saturday-booth">Saturday: Booth</h2>
<p>We had a standard blue Fedora booth with stickers, DVDs, 3D printer and (which was a bit of an experiment) blue soft drinks.</p>
<p><img src="https://fedoraproject.org/w/uploads/7/79/PyConCZ_1.jpg" alt="Fedora booth at PyCon CZ 2015" /></p>
<p>The blue drinks were quite an attraction, but we had <em>a lot</em> of them and we were unsure whether people would drink them all, so we stepped up and started delivering the drinks to presentation rooms. We were distributing them during breaks and we were telling people that they’re a gift from Fedora community - the responses were great and it attracted even more people to our booth. We almost run out of the drinks on Saturday and only a few bottles remained for Sunday workshop sessions. See the cases behind Slávek, our “Python 3 terrorist”, as some call him:</p>
<p><img src="https://fedoraproject.org/w/uploads/4/46/PyConCZ_2.jpg" alt="Fedora booth at PyCon CZ 2015" /></p>
<p>If you observe the photo carefully, you might find some new Fedora Loves Python stickers we’ve designed for this event (we still have plenty more for upcoming events).</p>
<h2 id="sunday-python-3-porting-sprint">Sunday: Python 3 porting sprint</h2>
<p>On Sunday, the sprints took place at a different venue and there was no room for booth, so we changed our tactics and organized a sprint instead.
<a href="https://fedoraproject.org/wiki/FAD_Python_3_Porting_2015">Python 3 Porting FAD</a> was an international Fedora Activity Day focused on porting stuff to Python 3; on PyCon we were there to do our part and to help others join Python 3 porting - either upstream or downstream in Fedora. While the FAD itself was a huge success, we must admit that we expected more PyCon attendants to be interested in our sprint. Lots of poeple visited workshops happening in the same time slot or just chatted with each other (drinking the rest of blue soft drinks). Still, we had a nice chat with several developers interested in Python 3 porting and we managed to get our “Python 3 is <em>the</em> Python” message across to quite a few people. Reactions to that were pretty great, the Python community really does appreciate where Fedora is moving.</p>
<p><em>Written with Slávek’s help.</em></p>
FAD Rheinland 2014: New ideas for outreach2014-12-09T00:00:00+00:00http://eng.hroncok.cz/2014/12/09/fad-rheinland-2014
<p>Last weekend, I’ve attended <a href="https://fedoraproject.org/wiki/FAD_Rheinland_2014">FAD Rheinland 2014</a>, the annual event where Fedora Ambassadors from EMEA meet and discuss the budget for next year, swag ideas and most importantly (at least form my POV), what to do differently to bring more users to Fedora.</p>
<p>Apart form meeting old and new friends, having fun together and discovering a fantastic board/card game called <a href="http://en.wikipedia.org/wiki/The_Saboteur">The Saboteur</a> (go get it now!), there were some ideas I want to share with others.</p>
<p>(I’m writing this on a train from Brno to Prague, so expect a lot of typos.)</p>
<h2 id="focus-on-different-events-bring-non-ambassadors-from-fedora">Focus on different events, bring non-Ambassadors from Fedora</h2>
<p>For years, we’ve been attending events like FOSDEM, LinuxTag and other Linux and free software specific conferences and meet-ups. As Fedora we need to have a booth on such events and we’ll have to continue to be there, to show others Fedora is still here. However, I don’t think (and there were others with the same opinion), that we’ll get any new users to Fedora doing it. Most of the visitors there already have their favorite Linux distro and I don’t think that we can do much on the booth that would convince them change their minds.</p>
<p>Based on Fedora.NEXT target audience (have a look at Christoph’s slides (I’ll add link ASAP)), we’ve agreed that we have to visit other types of events as well. General developer conferences (focusing on some topic, such as Python, Perl, Android, …), maker events (promote 3D printing in Fedora, etc.), barcamps, gaming cons. However, for most of us, this is stepping out of our comfort zone. Most Ambassadors are not experts in those fields. You cannot expect a random Fedora Ambassador to go to a Python dev meet-up in his/hers area and speak about why Fedora is good for Pythonists (well, we have dome ambassadors who could, but not all of them).</p>
<p>So we would like to “reach in” for people who knows their stuff and bring them to events to promote Fedora. The idea is something like:</p>
<ol>
<li>Bob, the Fedora Ambassador, finds out there’s an event related to XYZ topic in his area and thinks we might promote Fedora there.</li>
<li>However, Bob knows nothing (or little) about XYZ, but finds out in Fedora, we have a working group or SIG that focuses on XYZ or related stuff, so Bob writes an e-mail to their mailing list informing about the event and saying he wants a booth personal or speaker. Bob can allocate money to pay for the trip, accommodation and entrance fee if necessary.</li>
<li>Anna is a Fedora contributor, who focuses mainly on XYZ and lives near the area. She would love to go, but she has no idea how to run a booth or how to get sponsored form Fedora. Luckily Bob can help her.</li>
<li>Both Anna and Bob go to the event. Anna can promote XYZ in Fedora, Bob can answer general questions about Fedora Anna might not know the answer.</li>
</ol>
<p>We think this might work well, because a lot of people form inside Fedora would love to go to such events, but does not have money to afford it and has no idea how would Fedora pey for them. The problem might be, how to find out what group form Fedora focuses on XYZ, but Bob can ask on the Ambassadors mailing list for help and some other Ambassadors might know.</p>
<p>Some of my ideas for events in the Czech Republic:</p>
<ul>
<li>3dexpo (3D printing event, we’ve been there, but Vojta on the booth is no expert in 3D printing)</li>
<li>barcamp Brno, Hradec Králové, Ostrava…</li>
<li>aDev Meetup (Android development, promote DevAssistant)</li>
<li>Pyvo (Python meetups, we have Fedorains there, but we don’t promote Fedora)</li>
<li>Model Hobby (a gigantic plastic modellers convention)</li>
</ul>
<h2 id="stop-producing-dvds-and-produce-flyers-instead">Stop producing DVDs and produce flyers instead</h2>
<p>DVDs, at least in Europe, are obsolete. I don’t have a DVD drive. You don’t have it. The users don’t have it. Or at least they won’t, soon. We should stop producing DVDs with Fedora. But we cannot just stop, we need a replacement.</p>
<p>There are visitors on events asking for DVDs. They collect them, or they take them and trash them later. If we say we have no DVDs (as sometimes, we just don’t, because there are no more), they are sad and go grab another DVD (with Ubuntu).</p>
<p>The idea was to produce Fedora USB sticks, but it much more expensive than DVDs. No way of producing so much sticks as we produce DVDs now. I’ve (together with others, including Jaroslav Řezník) had an idea to produce flyers that look like DVD sleeve - so we can put them on the boot and it looks like a DVD. If the cover is hard enough, the visitors might even think they are getting a DVD, unless they look inside. It should be a booklet shaped flyer with the following information:</p>
<ul>
<li>“Where is the DVD” section (describing DVDs are deprecated and are replaced with QR code)</li>
<li>“What is Fedora” section</li>
<li>Fedora products (Workstation, Server, Cloud) descriptions (and now we are not supposed to call this products anymore)</li>
<li>“What’s new in Fedora 22” (23, 24…)</li>
</ul>
<p>And alternatively have some flyers for specific events and have a page about XYZ. The flyers might also be localized.</p>
<p>The problems with flyers is, that they cannot look cheap. Otherwise the visitors might as well get the idea that we are doing this to save money. And we, the Ambassadors, cannot design them, we need the design team to do it. And as I’ve heard, there are open tickets in the design trac for flyers for years without a response :( Last propper flyer we had was Fedora Core 4 (or something like that). We had Fedora Cloud flyers in the recent past, but nobody seems to like them anyway.</p>
<p>I hope that the transfer <a href="https://fedorahosted.org/famsco/ticket/373">from FAmSCo to FOSCo</a> that covers Ambassadors, Design and Marketing will bring us easier cooperation.</p>
<h2 id="more">More…</h2>
<p>There has been more. You can see the <a href="https://titanpad.com/C13BPojw05">raw notes from the event</a> and there should be the meeting minutes, but i cannot find them. And don’t forget to get <a href="http://en.wikipedia.org/wiki/The_Saboteur">The Saboteur</a> :D</p>
State of Python 3 as default in Fedora2014-02-12T00:00:00+00:00http://eng.hroncok.cz/2014/02/12/python3-fedora-default
<p>First of all, if you don’t know anything about Python 3 in Fedora, go read <a href="https://fedoraproject.org/wiki/Changes/Python_3_as_Default">this</a> before you go any further.</p>
<p>OK, now when you have read it (you have, right?), let me try to explain what needs to be done first and what’s blocking us. There is <a href="https://fedoraproject.org/wiki/User:Churchyard/python3">this gigantic table of packages that need to be ported to Python 3</a> - that means ship both 2 and 3 subpackages for modules, or use Python 3 interpreter for apps. From that, you can see what packages need some love. But it doesn’t show what is important and what should be targeted first.</p>
<h2 id="cloud">Cloud</h2>
<p>You may be surprised that cloud is my first topic. The thing is, cloud images of Fedora need to be as small as possible. They simply cannot ship both Pythons. While in other use cases we might be able to ship Python 2 as well as Python 3 with the default installation, cloud is not that case. The most important thing within cloud is <a href="https://bugs.launchpad.net/cloud-init/+bug/1247132">cloud-init</a>. It has two major deps that need to be ported first: <a href="https://github.com/boto/boto/issues/677">boto</a> and templating system.</p>
<p>Boto is somehow ported to Python 3 in a <a href="https://github.com/kurin/boto/tree/py3kport">GitHub repo by one individual</a> and it doesn’t follow upstream releases. It also requires an old version of HTTPretty that somehow did work with Python 3 (probably by accident). However, <a href="https://github.com/gabrielfalcao/HTTPretty/pull/143">HTTPretty now gets Python 3 support in newer versions</a>. So in some time, this might be solved by upstream (if we ask them politely once in a while).</p>
<p>cloud-init templating system is the real problem. Cloud-init uses <a href="http://www.cheetahtemplate.org/">Cheetah</a> and that is very old and probably dead, latest release is from 2010 and the documentation talks about Python 2.3. Porting that thing to Python 3 is probably not an option (and if you don’t think so, look at the code). So for cloud-init to be ported to Python 3 and for other reasons, <a href="https://bugs.launchpad.net/cloud-init/+bug/1219223">changing the templating system was proposed</a>. However, upstream wants 100 % backward compatibility with Cheetah templates and that’s not gonna happen magically. So adding a compat layer between some other templating system, that has a similar syntax as Cheetah (such as <a href="http://docs.makotemplates.org/en/latest/usage.html#basic-usage">Mako</a>) would help a lot. Good starting point for you, if you want to push thing forward.</p>
<h2 id="yum-dnf-and-friends">Yum, DNF and friends</h2>
<p>Yum won’t be ported to Python 3. We’ll ship DNF instead (maybe renamed to Yum). DNF should run with Python 3 just fine, but we’ll have to wait until it’s suitable as default. We cannot drop Python 2 from default installation while having Yum in as a default installer. According to upstream, <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1014559#c3">PackageKit should be Python 3 compatible when it drops Yum backend</a>, but as we see it, there are still some <a href="https://bugs.freedesktop.org/show_bug.cgi?id=66992">things that need to be ported</a>.</p>
<h2 id="anaconda">Anaconda</h2>
<p>And I’d say this is the last big thing that needs to be ported before we can drop Python 2 from default installation. Anaconda has a long list of dependencies (most of them are part of Anaconda project) that need to be ported. Anaconda developers are slowly working towards Python 3, but they have lots of other tasks on their shoulders and helping them would be nice. <del>Porting <a href="https://bugzilla.redhat.com/show_bug.cgi?id=984907">authconfig</a> looks like a good place to start as it’s developer clearly stated that he has no time to do it.</del> (The last sentence is probably <a href="https://lists.fedoraproject.org/pipermail/devel/2014-February/195466.html">false</a>.)</p>