opensource.com

3 tips to manage large Postgres databases

Tue, 02/28/2023 - 16:00

3 tips to manage large Postgres databases elizabeth.chri… Tue, 02/28/2023 - 03:00

The relational database PostgreSQL (also known as Postgres) has grown increasingly popular, and enterprises and public sectors use it across the globe. With this widespread adoption, databases have become larger than ever. At Crunchy Data, we regularly work with databases north of 20TB, and our existing databases continue to grow. My colleague David Christensen and I have gathered some tips about managing a database with huge tables.

Big tables

Production databases commonly consist of many tables with varying data, sizes, and schemas. It's common to end up with a single huge and unruly database table, far larger than any other table in your database. This table often stores activity logs or time-stamped events and is necessary for your application or users.

Really large tables can cause challenges for many reasons, but a common one is locks. Regular maintenance on a table often requires locks, but locks on your large table can take down your application or cause a traffic jam and many headaches. I have a few tips for doing basic maintenance, like adding columns or indexes, while avoiding long-running locks.

Adding indexes problem: Index creation locks the table for the duration of the creation process. If you have a massive table, this can take hours.

CREATE INDEX ON customers (last_name)

Solution: Use the CREATE INDEX CONCURRENTLY feature. This approach splits up index creation into two parts, one with a brief lock to create the index that starts tracking changes immediately but minimizes application blockage, followed by a full build-out of the index, after which queries can start using it.

CREATE INDEX CONCURRENTLY ON customers (last_name)Adding columns

Adding a column is a common request during the life of a database, but with a huge table, it can be tricky, again, due to locking.

Problem: When you add a new column with a default that calls a function, Postgres needs to rewrite the table. For big tables, this can take several hours.

Solution: Split up the operation into multiple steps with the total effect of the basic statement, but retain control of the timing of locks.

Add the column:

ALTER TABLE all_my_exes ADD COLUMN location text

Add the default:

ALTER TABLE all_my_exes ALTER COLUMN location SET DEFAULT texas()

Use UPDATE to add the default:

UPDATE all_my_exes SET location = DEFAULT Adding constraints

Problem: You want to add a check constraint for data validation. But if you use the straightforward approach to adding a constraint, it will lock the table while it validates all of the existing data in the table. Also, if there's an error at any point in the validation, it will roll back.

ALTER TABLE favorite_bands ADD CONSTRAINT name_check CHECK (name = 'Led Zeppelin')

Open source and data science What is data science? What is Python? Data scientist: A day in the life Try OpenShift Data Science MariaDB and MySQL cheat sheet Latest data science articles

Solution: Tell Postgres about the constraint but don't validate it. Validate in a second step. This will take a short lock in the first step, ensuring that all new/modified rows will fit the constraint, then validate in a separate pass to confirm all existing data passes the constraint.

Tell Postgres about the constraint but do not to enforce it:

ALTER TABLE favorite_bands ADD CONSTRAINT name_check CHECK (name = 'Led Zeppelin') NOT VALID

Then VALIDATE it after it's created:

ALTER TABLE favorite_bands VALIDATE CONSTRAINT name_checkHungry for more?

David Christensen and I will be in Pasadena, CA, at SCaLE's Postgres Days, March 9-10. Lots of great folks from the Postgres community will be there too. Join us!

Try these handy solutions to common problems when dealing with huge databases.

Image by:

Internet Archive Book Images. Modified by Opensource.com. CC BY-SA 4.0

Data Science Databases What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Delegate common tasks with an open source automation tool

Tue, 02/28/2023 - 16:00

Delegate common tasks with an open source automation tool lnxchk Tue, 02/28/2023 - 03:00

A key precept for site reliability engineering (SRE), platform engineering, and *Ops teams is automating non-essential toil. Think of toil as any tasks that must be done for systems to run smoothly but don't directly impact functionality. Tasks like running security updates and installing or upgrading packages fall into the toil category.

As teams become more proficient at creating automation, automating for other teams becomes more appealing. Delegating tasks to other technical teams so they can self-serve saves time on both sides of a ticket request: The requesting teams get tasks done when they need them, and the implementing teams aren't buried under a ticket queue full of requests that might be simple but need to be done correctly.

Automation should address three gaps to reap these benefits:

Knowledge: Tasks can have many steps and dozens of possible options.
Skills: Modern ecosystems consist of an enormous number of components with specific execution and configuration requirements.
Access: Environments are restricted to a few users for security and compliance reasons.

Powerful automation platforms give teams a solution for these three gaps, providing a centralized resource to create, maintain, and grant access to automation for many types of tasks. These tools shouldn't ask too much of the requestor, who might know, "I want a developer environment in our cloud account" and not, "I want a set of containers built with this image, networked in that way, with access to the test data."

Rundeck is an open source software used to quickly automate manual tasks. Use it to create workflows consisting of commands, scripts, and APIs. These workflows might include software management, configurations, and scheduled events. Rundeck logs these activities for transparency and troubleshooting. How can Rundeck address the example of a developer needing an environment in the organization's cloud? Consider the following points.

Capture and share what you know

Creating automation that "civilians" can use has some challenges. The folks in your organization who work on other teams have their own skill sets and expertise, and it's not uncommon for different teams to have diverse skill sets and even platforms or environments where they are comfortable.

Administrators can automate a wide variety of tasks to be consumed by other teams in the organization. Common targets include provisioning environments and deploying new software versions, but runtime tasks such as service restarts and capacity adjustments are also helpful. These tasks might contain multiple steps, require access to various systems, platforms, or dashboards, and need to adhere to institutional requirements.

A script or other tool represents the team's knowledge of those tasks and others. The solution might be a simple shell script or a more elaborate Python or Go tool; it could run a few commands or a series of requests to a third-party API. Sysadmins should be able to write automation tools in the languages and platforms that make the most sense for the job. Rundeck supports a wide variety of tools and functionality.

Delegate safely

Internal tools aren't usually meant for general consumption, so opening up access to other teams can be challenging. These tools were designed and developed with a specific persona in mind—that of the people who created them! If you want to delegate work to others, you must consider what knowledge they bring and what you can do with that knowledge.

Automation with open source Download now: The automated enterprise eBook Free online course: Ansible essentials Ansible cheat sheet eBook: A practical guide to home automation using open source tools A quickstart guide to Ansible eBook: 7 examples of automation on the edge More articles about open source automation

In the example of a developer who wants a new development environment, you already know things about the developer, like their team or department and the projects they can access. You also know how work gets done for those teams or projects. If the developer is on Team A, and you already have a script called build_dev_env_TeamA.sh, then you can delegate that job to the developer with an automation platform like Rundeck.

With Rundeck, the experts on your teams can use the scripts they already have to create jobs and plugins that others in the organization can utilize. When developers log into the Rundeck server, they only see the jobs and resources they've been granted access to. In this example, that might be a job named "Build a Developer Environment - Team A". The developer is satisfied, and your experts can continue their work uninterrupted. Rundeck records the activity and includes the execution of the job in the audit log. Should something go wrong, it can be found and fixed.

Give everyone what they want: Time

Teams reap the most rewards from automation that saves them time. A purpose-built automation platform like Rundeck gives time back on both sides of complex tasks: Requestors no longer wait for someone else to execute a task for them, and the experts keep their time working on other projects. The organization gains insight into the jobs run in the environment via the audit logs, and everything is secured using existing authorization solutions and Rundeck's access control lists features.

For more on the open source Rundeck platform, check out the tutorials and documentation and download it today.

Rundeck is an open source software used to quickly automate manual tasks. Use it to create workflows consisting of commands, scripts, and APIs.

Image by:

opensource.com

Automation Sysadmin DevOps What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Essential tips and tricks for your first tech job

Mon, 02/27/2023 - 16:00

Essential tips and tricks for your first tech job ftaj Mon, 02/27/2023 - 03:00

First days at work are scary. I still recall many instances where I lay awake at night before my first day at work, having an internal meltdown over what would happen the next day. Starting a new job is uncharted territory for most people. Even if you're a veteran in the industry, there's no denying that there can be a part of you that's a bit terrified of what is to come.

Understandably, a lot is happening. There are new people to meet, new projects and technologies to understand, documentation to read, tutorials to sit through, and endless HR presentations and paperwork to fill out. This can be overwhelming and, coupled with the considerable degree of uncertainty and unknowns you're dealing with, can be quite anxiety-inducing.

Two reasons motivated me to write about this subject. The first one being that back when I was a student, most of the discussion revolved around getting a job in tech, and no one talked about what happened next. How do you excel in your new role? Now that I look back, I think I assumed that the hard part is getting the job, and whatever comes after, I could probably figure out myself.

Similarly, once I started working in the industry, most of the career-related content I came across was about how to go from one senior level to another. No one really talked about what to do in the middle. What about the interns and the junior engineers? How do they navigate their early careers?

After completing three years of full-time professional experience as a software engineer (and a couple of internships before), I reflected on my time. I put together a list of tips and tricks I've employed while settling into a new tech role. I wanted to look beyond just the first couple of months and prioritize helping achieve long-term success.

Reflect on existing processes and documentation

Most new employees start by either having a ton of documentation thrown their way or none at all. Instead of being overwhelmed by either of these possibilities, you could view this as an opportunity.

Identify gaps in existing documentation and think about how you could improve it for the next engineer that gets onboarded. This not only shows initiative on your part but also demonstrates that you're committed to improving existing processes within your team.

I've seen both ends of the spectrum. I've been on teams with no documentation whatsoever. I've also been on teams that were very diligent with keeping their documentation up to date. Your path is pretty straightforward with the former, and you can work on creating that missing documentation. With the latter, you can always think of ways to improve what already exists. Sometimes, too much documentation in written form can also feel intimidating, especially for new employees. Some things might be better explained through other mediums, like video tutorials or screencasts.

Ask questions

I encourage you to look into whether a buddy will be assigned to you when you're starting. This is a fairly common practice at companies. The purpose of a buddy is to help you as you are onboarded. I've found this incredibly helpful because it gives you someone to direct all your questions, and you don't have to run around trying to find the right person/team.

While asking questions should always be encouraged, it is also necessary to do your homework before you ask those questions, including:

Do your research. This encompasses doing a web search, checking forums, and reading existing documentation. Use all the available tools at your disposal. However, it is essential to timebox yourself. You must balance doing your due diligence and keeping project deadlines and deliverables in mind.
Talk it out. As someone whose first language isn't English, I recommend talking things out loud before asking questions. In my experience, I've often found that, especially when I'm struggling with something difficult, I think in one language (probably my native language) and must explain it in another. This can be a bit challenging sometimes because doing that translation might not be straightforward.
Organize your thoughts. When struggling with something, it's very common to have many scrambled ideas that make sense to us but might not necessarily make sense to another person. I suggest sitting down, gathering your thoughts, writing them down, and talking through them out loud. This practice ensures that when you're explaining your thought process, it flows as intended, and the listener can follow your train of thought.

This approach is called the rubber duck technique, a common practice developers use while debugging. The concept is that sometimes explaining your problem to a third person can be very helpful in getting to the solution. This is also a testament to your excellent communication skills.

Respect people's time. Even if you're reaching out to someone like your buddy, be cognizant of the fact that they also have their day-to-day tasks to complete. Some things that I've tried out include the following:

Write down my questions and then set aside some time with my mentor so I could talk to them.
Compile questions instead of repeatedly asking for help so your mentor can get to them when they have time.
Schedule a quick 15-20 min video chat, especially if you want to share your screen, which is a great way to showcase your findings.

I think these approaches are better because you get someone's undivided attention instead of bothering them every couple of minutes when their attention might be elsewhere.

Deep dive into your projects

Even on teams with excellent documentation, starting your technical projects can be very daunting since multiple components are involved. Over time though, you will understand how your team does things. However, it can save you time and potential headaches to figure this out early on by keeping a handy list to refer to, including basic project setup, testing requirements, review and deployment processes, task tracking, and documentation.

If there's no documentation for the project you're starting on (a situation I have been in), see if you can identify the current or previous project owner and understand the basic project structure. This includes setting it up, deploying it, etc.

Identify your team's preference in the IDE (integrated development environment). You're free to use the IDE of your choice, but using the same one as your team can help, especially when debugging, since the choice of IDE impacts debugging. Different IDEs offer varying degrees of debugging support.
Understand how to do debugging, and I don't just mean using print statements (not that there's anything wrong with that approach). Leverage your team's experience here!
Understand testing requirements. This might depend on the scope of your project and general team practices, but the earlier you figure this out, the more confident you'll be in the changes you push to production.
Visualize the deployment process. This process can vary by team, company, etc. Regardless of how informal or formal it may be, make sure you understand how your changes get deployed to production, what the deployment pipeline looks like, how to deploy changes safely, what to do in case of failed builds, how to rollback faulty changes, and how to test your changes in production.
Understand the ticketing process. Understand how to document tickets and the level of detail expected. You will see a lot of variation here. Some companies expected us to submit our tickets daily, showing our progress. Other companies might not require that level of documentation.

Given everything I just mentioned, a beneficial, all-in-one exercise you can do in the first couple of weeks is to shadow another engineer and do peer coding sessions. This allows you to observe the entire process, end to end, from the moment a ticket is assigned to an engineer to when it gets deployed to production.

The first couple weeks can also feel frustrating if you're not yet given an opportunity to get your hands dirty. To counter this, ask your manager to assign some starter tickets to you. These are usually minor tasks like code cleanup or adding unit tests. Still, they allow you to tinker with the codebase, which helps improve your understanding and gives you a sense of accomplishment, which is a very encouraging feeling in the early days of a new job.

Speak up, especially when you're stuck

I want to stress the importance of communication when you're stuck. This happens, especially in the early months of a new job, and as frustrating as it can be, this is where your communication skills will shine.

Be transparent about blockers and your progress. Even if it's something as trivial as permission issues (a fairly common blocker for new employees), ensure that your manager is aware.
Don't wait until the last day to report if something will be delayed. Delays in your project push many other things forward. Share necessary project delays well in advance, so your manager can share this with stakeholders.
Don't forget things like thoroughly testing your changes or documenting your code just because you're in a rush.

Gain technical context

Gaining technical context is something I've personally struggled with, and I've actively worked on changing my approach in this area.

When I started as an intern, I would go in with a very focused mindset regarding what I wanted to learn. I'd have a laser-sharp focus on my project, but I'd completely turn a blind eye to everything else. Over the years, I realized that turning a blind eye to other or adjacent projects might not be the wisest decision.

First and foremost, it impacts your understanding of your work. I was naive to think I could be a good engineer if I focused exclusively on my project. That's just not true. You should take the time to understand other services with which your project might interact. You don't need to get into the nitty gritty, but developing a basic understanding goes a long way.

A common experience that new employees undergo is disconnecting from the rest of the company, which is a very natural feeling, especially at larger companies. I'm someone who develops a sense of exclusion very quickly, so when I moved to Yelp, a significantly larger company than my previous one, with projects of a much larger scale, I prioritized understanding the big picture. Not only did I work on developing an understanding of my project but also of other adjacent projects.

In my first few weeks at Yelp, I sat down with various engineers on my team and asked them to give me a bird's eye view of what I would be doing and the project's overarching goal. This approach was incredibly helpful because not only did I get varying degrees of explanations based on how senior the engineer was and how long they had been working on the project, but it also deepened my understanding of what I would be working on. I went into these meetings with the goal that my knowledge of the project should allow me to explain what I do to a stranger on the street. To this end, I asked my tech lead to clarify at what point my work came into the picture when a user opened the Yelp app and searched for something.

Architecture diagrams can also help in this scenario, especially when understanding how different services interact.

Establish expectations

For the longest time, I thought that all I needed to do was my best and be a good employee. If I was doing work, meeting goals, and no one complained, that should be good enough, right? Wrong.

You must be strategic with your career. You can't just outsource it to people's goodwill and hope you'll get the desired results just because you're meeting expectations.

Establish clear criteria the moment you start your new job. This varies by company, as some organizations have very well-defined measures while others might barely have any. If it's the latter, I suggest you sit down with your manager within the first couple of weeks and establish and unanimously agree on a criterion.
Make sure you thoroughly understand how you will be evaluated and what measures are used.

I remember walking out of my first evaluation very confused in my first full-time role. The whole conversation had been very vague and hand-wavy, and I had no clarity about my strengths, weaknesses, or even steps to improve.

At first, it was easy to attribute everything to my manager because the new employee in me thought this was their job, not mine. But over time, I realized that I couldn't just take a backseat as far as my performance evaluations were concerned. You can't just do good work and expect it to be enough. You have to actively take part in these conversations. You have to make sure that your effort and contributions are being noticed. From regularly contributing to technical design conversations to setting up socials for your team, ensure that your work is acknowledged.

Tying into establishing expectations is also the importance of actively seeking feedback. Don't wait until your formal performance evaluations every three or four months to find out how you're doing. Actively set up a feedback loop with your manager. Try to have regular conversations where you're seeking feedback, as scary as that may be.

Navigate working in distributed teams

The workplace has evolved over the past two years, and working in remote and distributed teams is now the norm instead of a rarity. I've listed some tips to help you navigate working in distributed teams:

Establish core hours and set these on your calendar. These are a set of hours that your team will unanimously agree upon, and the understanding is that everyone should be online and responsive during these hours. This is also convenient because meetings only get scheduled within these hours, making it much easier to plan your day.
Be mindful of people's time zones and lunch hours.
In the virtual world, you need to make a greater effort to maintain social interactions, and little gestures can go a long way in helping make the work environment much friendlier. These include the following:
- When starting meetings, exchange pleasantries and ask people how their weekend/day has been. This helps break the ice and enables you to build a more personal connection with your team members, which goes beyond work.
- Suggest an informal virtual gathering periodically for some casual chit-chat with the team.

More open source career advice Open source cheat sheets Linux starter kit for developers 7 questions sysadmins should ask a potential employer before taking a job Resources for IT artchitects Cheat sheet: IT job interviews Maintain a work-life balance

At the beginning of your career, it's easy to think that it's all about putting in those hours, especially given the 'hustle culture' narrative that we're fed 24/7 and the idea that a work-life balance is established in the later stages of our careers. This idea couldn't be further from the truth because a work-life balance isn't just magically going to occur for you. You need to actively and very diligently work on it.

The scary thing about not having a work-life balance is that it slowly creeps up on you. It starts with you checking emails after hours and then slowly makes its way to you, working over weekends and feeling perpetually exhausted.

I've listed some tips to help you avoid this situation:

Turn off/pause notifications and emails and set yourself to offline.
Do not work weekends. It starts with you working one weekend, and the next thing you know, you're working most weekends. Whatever it is, it can wait until Monday.
If you're an on-call engineer, understand your company's policies surrounding that. Some companies offer monetary compensation, while others may give time off in lieu. Use this time. Not using your benefits like PTO (paid time off) and wellness days really shortens your longevity at work.

Wrap up

There's no doubt that starting a new job is stressful and difficult. I hope that these tips and tricks will make your first few months easier and set you up for great success with your new position. Remember to communicate, establish your career goals, take initiative, and use the company's tools effectively. I know you'll do great!

Starting a new job is daunting for anyone. Here's how to navigate the early days at your first tech job.

Image by:

Opensource.com

Careers SCaLE What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

AI robot wrestling with open source

Sat, 02/25/2023 - 16:00

AI robot wrestling with open source omichel Sat, 02/25/2023 - 03:00

The ICRA 2023 conference will take place from May 29 to June 2, 2023 in London. It brings together the world’s top robotics researchers from academia and industry. During this event, several competitions are organized to foster new achievements in robotics. Among them, the wrestling competition will confront the best algorithms controlling intelligent humanoid robots in a wrestling game. Although the competition takes place in simulation, these little 3D guys are true digital twins of existing real robots named NAO. So their AI could transfer into the body of their real counterparts in a pretty straightforward way.

Image by:

(Olivier Michel, CC BY-SA 4.0)

Digital twin of a real robot

Each robot is equipped with a number of sensors and motors. These devices are available from the programming interface. The controller program can retrieve their measurements, such as camera images, process them, and send motor commands to drive the robot on its way to floor its opponent. Each participant can submit several versions of its controller program. For each new submission, a series of games is run in the cloud. Everyone can view them immediately as 3D animations from the web page of the leaderboard.

All shots allowed

The goal for the little guys on the ring is to floor their opponent in less than three minutes. If none succeed, the robot who covered the largest area wins. During the game, all shots are allowed, including using a stick or throwing a little plastic duck. Because these guys are sometimes so clunky, watching the games can be very entertaining.

Image by:

(Olivier Michel, CC BY-SA 4.0)

100% open source software

The complete competition infrastructure runs on GitHub and fully relies on open source software, including the Webots robot simulator. The participant workflow is straightforward for any software developer and can be easily re-used to implement a similar competition.

Image by:

(Olivier Michel, CC BY-SA 4.0)

Open to anyone

Although it requires some basic skills in software development, participation is open to anyone. The robots can be programmed in Python, Rust, C, C++, Java or ROS. There’s no limitation on the software libraries or programming languages that can be used.

Save the date

If you want to participate, you can register until May 23rd, 2023. However, you’re encouraged to register as early as possible to increase your chance of success.

January 16th, 2023: Registration opens and qualification games start
May 23rd, 2023: Selection of the best 32 teams
May 30th, 2023: 1/16 finals
May 31th, 2023: 1/8 finals
June 1st, 2023: 1/4 finals
June 2nd, 2023: Semifinals, third place game, and final

The finals will take place during the ICRA 2023 conference in London, and will be broadcasted online in real time. Remote participation will be possible.

Image by:

(Olivier Michel, CC BY-SA 4.0)

Open science and sustainability Video series: ChRIS (ChRIS Research Integration System) Explore Red Hat Research projects 6 articles to inspire open source sustainability How Linux rescues slow computers (and the planet) Latest articles about open science Latest articles about open education Latest articles about sustainability Grand prize

The winner of the competition will receive one Ether crypto-currency. Although its value was USD 1,659 on February 3rd 2023, it may be completely different on June 2.

Fostering the development of open science

The competition is organized by Cyberbotics Ltd., a Swiss-based company developing the open source Webots robot simulator. It aims at promoting the use of open source software tools, including cloud-based simulations, to achieve incremental progress in robotics research. By relying on open source software, running in virtual machines in the cloud, research results in robotics become easily reproducible by anyone. This allows researchers to build their own research on top of the results achieved by their colleagues, paving the way to incremental research in the spirit of open science.

Here are relevant links for participation:

Hope to see you there!

Program the artificial intelligence (AI) of simulated robots in an online international competition.

Image by:

Opensource.com

AI and machine learning Science Hardware What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

5 open source tools to take control of your own data

Fri, 02/24/2023 - 16:00

5 open source tools to take control of your own data wsamh Fri, 02/24/2023 - 03:00

Back in the old days, there was no cloud. Everything was on your phone. Maybe you had a microSD card that you backed up everything on. Eventually, the SD card would stop working, and you lost everything unless you'd saved it on a writable CD or DVD or stored it on your PC. Self-hosting was tough in those days, and it was expensive. Software wasn't as accessible as it is now.

Today, it's common for phones not to have an SD card slot. The good news is that software is good enough that you can back up everything you own on a single Raspberry Pi, spare laptop, or mini-PC.

You can own your own data and data stack by self-hosting. Containers and personal cloud software make it possible. In this article, I share several of my favorite ways to make that happen.

Linux Containers What are Linux containers? What is Kubernetes? Free online course: Deploy containerized applications eBook: A guide to Kubernetes for SREs and sysadmins Free online course: Running containers with Red Hat technical overview Podman cheat sheet The latest articles on Linux containers Containers

A container is software consisting of everything required for an application to work. Each container acts as its own computer and doesn't affect other containers or software on your host server. With this technology, you can keep your software up to date without breaking your system. It also allows you to control where data is stored, making backing up your data easy.

Learning to use containers can be intimidating. I started with Docker, although many other container engines exist, including Podman and Istio. It didn't take long for me to get the hang of it. I found that containers make self-hosting services easier than ever. If you're familiar with installing applications on the Linux terminal, you'll get the hang of it quickly.

Syncthing

One of the easiest ways to back up your data is through Syncthing. This open source software synchronizes data across different devices. Select the folder you want to exist on two (or more) devices, and then that data and any changes to it are reliably kept updated on each device.

This isn't just a convenient way to share data; it's also a backup scheme. Should one hard drive go down, you have a copy of your important data on another device. Once you restore the broken PC, you can reconnect with Syncthing, and it synchronizes everything you lost. Syncthing is useful for storing data on multiple devices in different locations, including on machines outside your house (at a friend or family member's home, for instance). It's also a great off-site backup tool.

Nextcloud

Nextcloud is an open source alternative to Google Drive or Dropbox. It's also multi-user, so once you install Nextcloud, you can set up distinct logins for each user. There are a variety of Nextcloud apps for phones and PCs. You can auto-synchronize your photos and then view photos from the app or a web browser. You can mark files public to share them with the rest of the internet.

Similar to Syncthing, a client can also synchronize files between your server and your desktop or laptop. Nextcloud also has components to let you manage contacts and calendars, and of course, you can synchronize them between other devices.

In fact, you can install many kinds of apps on Nextcloud, including programs to store notes, manage email, chat with others, and more. The Nextcloud environment includes an "app store" of open source applications.

Jellyfin

If you're interested in managing your own media server, then you're in luck. Jellyfin takes your media, like movies, TV shows, and music, and makes them available to any device you allow access. You can use Jellyfin to scrape the web for metadata, automatically retrieving cover art and media information.

Jellyfin also works without the internet. When your internet goes out and you can't connect to your favorite streaming service, you can use your local network to connect to your Jellyfin server and watch or listen to your media. I have had this happen, and I can attest that it's a great way to keep yourself and your family entertained.

Home server

These are just a few services you can install on any Linux PC or laptop. You need a server that's always on to ensure your services are constantly available. That doesn't necessitate a major investment, though. You can use many kinds of computers as Linux servers. The easiest and most inexpensive is a Raspberry Pi, which has excellent support with a helpful and enthusiastic community.

Getting a Raspberry Pi set up is "as easy as pie," thanks to the Raspberry Pi imager. It only needs about 5W of power, so it doesn't take much energy to keep it running. There are many similar low-powered devices, including the Odroid, Orange Pi, and Rockpi.

You can also install Linux on any PC or laptop and run it as a server. It's a great way to repurpose old computers.

Finally, you could use a Virtual Private Server (VPS). A VPS is a "slice" of space on a server located in a big data center. You pay rent on the server space and maintain it as you wish.

Your own data

When you put data on the cloud, it can be used without your control or consent. It may even be used without your knowledge. I don't foresee that issue improving.

We don't need private companies handling our data anymore. You can often replace corporate services to reduce the amount of data you're giving away.

In my opinion, we should all own our data, and we need to do it correctly, with open source. We can host services for personal use and for family and friends. I synchronize my calendar and contacts with my personal server (a Raspberry Pi in my home). I think it's worth fighting for, and there's no better time than right now.

Take your data out of the hands of proprietary corporations and into your own hands with open source solutions.

Image by:

Opensource.com

Containers Nextcloud Linux Raspberry Pi SCaLE What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Learn Lua with our new downloadable guide

Thu, 02/23/2023 - 16:00

Learn Lua with our new downloadable guide sethkenlon Thu, 02/23/2023 - 03:00

Lua is a programming language designed for simplicity and performance, used by video game and multimedia companies as a front-end scripting language. It's also used by the Awesome window manager, the Far file manager, the Howl text editor, and many more open source projects for its clarity and clean design. Lua is embeddable, too, so you can include Lua code in codebases of another language (such as Java, C, and C++), as well as interact with a rich C API. Whether you want to want to learn Lua to get into the gaming and media industry, or you're just interested in an easy scripting language with no upper limit, Lua is an approachable and powerful programming language.

Install Lua

On Linux, you can install Lua using your package manager. Your distribution may offer old versions of Lua in the interest of backward-compatibility. Install the latest version unless you have a reason to install an old version.

The Luau (with a trailing "u") programming language is a popular fork of Lua 5.1 created by Roblox. If you're using Luau or Lua for Roblox programming, install Lua 5.1.

On Windows or macOS, visit LuaDist.
Alternately, for macOS, use Homebrew.

Getting started with Lua

You write Lua code in a text editor. Don't use a word processor or office application. Use a plain old text editor or IDE, because those save text as literal plain text without any special formatting characters that tend to confuse the programs that need to interpret your code.

Most programs, Lua code included, can be broadly organized into these three sections:

Set up the environment: In this section, you identify the components you want your program to have available. In some programs, the components are just letters and numbers, while in advanced coding they may also include graphics and sound.
Build the engine: In this section, you tell Lua what to do with the components you've provided. In rudimentary programs, data is processed and an answer is produced. In advanced coding, this section runs in a loop while the user interacts with the application.
Go: Your code does nothing unless you tell it to start.

Suppose you want to write a simple Lua script that calculates the chance of a zombie apocalypse (or at least, that produces an arbitrary number you can claim is the chance of a zombie apocalypse). Getting an arbitrary number from a computer is difficult:

-- setup math.randomseed(os.time()) myNumber = math.random(0,100) -- engine function calculate() print("There is a " .. myNumber .. "% chance of a zombie apocalypse today.") end -- go calculate()

In the setup section, you start a random number generator with math.randomseed. As the source of randomness, you use the time as reported by your operating system. These are built-in functions of Lua. Nobody starts out knowing all the built-in functions of a programming language, so you find out about them from articles like this one, through task-based Internet searches, and by reading the Lua documentation.

Next, you create a variable. I often prefix variable names with "my" as a reminder to myself that it's something that I created for this application, not something built into Lua. A variable is a changeable (or "mutable," in programming terminology) value. You're claiming space in your computer's RAM and storing data there so you can use it later. In this case, the variable you create uses the math.random function of Lua to select a number between 0 and 100. Once again, this is a built-in function.

In the engine section, you create your own function, which I call calculate. There's nothing special about the term calculate except that it makes sense in the context of this application. You can name your functions practically anything. Whatever you call it, this is the part of your code that actually does something. Most computer programs have several functions, but this is a simple application, so it has just the one. The calculate function uses the built-in Lua print function to display text on the screen. As you can probably guess, two dots (..) in Lua indicate a continuation of a statement, similar to what an ellipsis does in English.

Finally, the go section runs the calculate function. If you forget to execute a function, then your application never runs. It would be like sitting in a car without turning the ignition.

The section labels setup, engine, and go are comments. Comments in Lua are indicated by two leading dashes (--), and it's a way for you to expressly tell Lua to ignore that line. In other words, setup, engine, and go aren't Lua keywords, and you don't need them in your code. They're just a useful way to remember how to structure your code.

Try running your application a few times:

$ lua ./zombie.lua There is a 78% chance of a zombie apocalypse today. $ lua ./zombie.lua There is a 10% chance of a zombie apocalypse today.Conditionals

At a very low level, computers operate according to binary conditions. An electrical signal is either present (1) or not (0). This manifests in code, too, and one of the classic methods modern programming languages provide to express that binary condition is the if-then statement.

An if-then statement causes the computer to analyze the data it has gathered so far, compare it to some arbitrary condition you define, and then take some action based on what it discovers. To make your first Lua application more dynamic, you can add an if-then statement to your function:

function calculate() print("There is a " .. myNumber .. "% chance of a zombie apocalypse today.") if myNumber > 50 then print("Take an umbrella!") else print("It's a good day for gardening!") end end

This if-then statement uses a little basic math to determine what the current run of the application is reporting. If there's more than a 50% chance of a zombie apocalypse, then some helpful advice is provided to the user. Else, the chance must be 50 or less, so different advice is provided. It's a simple condition that has the computer analyze the random number it has generated, and then take one or another action based on the result.

Run the code to see the results:

$ lua ./zombie.lua There is a 71% chance of a zombie apocalypse today. Take an umbrella! $ lua ./zombie.lua There is a 65% chance of a zombie apocalypse today. Take an umbrella! $ lua ./zombie.lua There is a 12% chance of a zombie apocalypse today. It's a good day for gardening!

There are many other kinds of conditional controls you can use in Lua, including repeat-until, while, for, and more.

Learning Lua

Nobody starts programming already knowing how to program. If you're interested in learning how to write code, Lua is a great way to start because it's a small language consisting mostly of conveniences to make your fledgling career as a programmer easier. It's not muddled up with lots of edge-case functions that you're not likely to use. Instead, Lua provides the building blocks you need to create your own unique functions that do exactly what you want them to do. Everything you need to know about Lua is available in the language's documentation, but if you want to continue a walk-through of the language, you can also download our eBook. By the end of the book, you'll know all the Lua basics, and where to find more Lua functions so you can move on to advanced Lua programming.

Whether you want to want to learn Lua to get into the gaming and media industry, or you're just interested in an easy scripting language with no upper limit, Lua is an approachable and powerful programming language.

Image by:

Ray Smith

Programming Download the eBook A guide to Lua This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

How I do automated accessibility testing for my website

Wed, 02/22/2023 - 16:00

How I do automated accessibility testing for my website dmundra Wed, 02/22/2023 - 03:00

This article covers adding accessibility tests to your site using Pa11y (pa11y-ci with axe) and Cypress (with cypress-axe) in GitLab CI/CD. I use a Jekyll website as an example, but any website technology that runs in CI/CD can leverage this setup.

Prep your website

In addition to getting your website to run in CI/CD, I recommend enabling an XML sitemap feature. A sitemap allows the accessibility tests to parse all URLs to find accessibility issues across the site. I recommend the jekyll-sitemap plugin for Jekyll sites.

Collecting a list of all major URLs is a good alternate step if a sitemap is not possible. The URLs should cover all potential layouts of the website, such as pages with the highest traffic or the most landings. This approach won't catch all accessibility issues, especially content level concerns, but it will test the layout and main pages.

This scenario requires the npm or yarn package managers. I used npm for this article. If your project doesn't have npm initialized, run the npm init command to create the package.json file.

Begin with Pa11y

Pa11y is a free and open source software that tests websites for accessibility issues. Pa11y-ci is the command line utility geared towards continuous integration (CI). Install pa11y-ci as a development dependency with npm:

$ npm i --save-dev pa11y-ci

After you complete the installation, edit the package.json and add the following commands to the scripts section:

"start-detached": "bundle exec jekyll serve --detach", "pa11y-ci:home": "pa11y-ci http://127.0.0.1:4000", "pa11y-ci:sitemap": "pa11y-ci --sitemap http://127.0.0.1:4000/sitemap.xml --sitemap-find https://accessibility.civicactions.com --sitemap-replace http://127.0.0.1:4000 --sitemap-exclude \"/*.pdf\""

start-detached: Starts the web server that will run Jekyll for testing.
pa11y-ci:home: Runs pa11y-ci tests on the home page. Useful for troubleshooting.
pa11y-ci:sitemap: Runs pa11y-ci tests using the sitemap and excludes PDFs. The sitemap will refer to the live site URLs, so replace those with local URLs for testing in the CI pipeline.

Add a JSON file named .pa11yci that configures pa11y-ci with various options. Here is a sample file:

{ "defaults": { "concurrency": 1, "standard": "WCAG2AA", "runners": ["axe", "htmlcs"], "ignore": [ "color-contrast", "frame-tested" ], "chromeLaunchConfig": { "args": ["--disable-dev-shm-usage", "--no-sandbox", "--disable-gpu"] }, "reporters": [ "cli", ["./pa11y-reporter-junit.js", { "fileName": "./pa11y-report-junit.xml" }] ] } }

concurrency: I reduced this set to 1 because increasing it caused errors (https://github.com/pa11y/pa11y-ci/issues/168 covers the bug, which might be fixed).
standard: I have stuck with the default WCAG2AA as the goal for this site.
runners: I ran axe (run tests using axe-core) and htmlcs (default, run tests using HTML CodeSniffer) to cover all potential accessibility issues.
ignore: With newer versions of axe and some of the changes to the site, I ran into color contrast false positives. I also have an embedded iframe that requires separate testing that axe will report about. I have follow-up issues to examine the axe results, so I am ignoring those criteria for now.
chromeLaunchConfig: pa11y-ci uses Chrome, and I found that the GitLab CI pipeline requires that the Chrome browser runs properly in the pipeline.
reports: I use the default command line reporter, but I also added a custom reporter that reports on the pa11y-ci results in a junit format. This came in handy for reporting the results in the GitLab CI pipeline.

That's it. Run this setup locally using npm, and you will see the following output (truncated for brevity):

dmundra in ~/workspace/accessibility/accessibility on branch main > npm run start-detached > start-detached > bundle exec jekyll serve --detach Configuration file: /Users/dmundra/workspace/accessibility/accessibility/_config.yml Source: /Users/dmundra/workspace/accessibility/accessibility Destination: /Users/dmundra/workspace/accessibility/accessibility/_site Incremental build: disabled. Enable with --incremental Generating... done in 8.217 seconds. Auto-regeneration: disabled when running server detached. Server address: http://127.0.0.1:4000 Server detached with pid '14850'. Run `pkill -f jekyll' or `kill -9 14850' to stop the server. dmundra in ~/workspace/accessibility/accessibility on branch main > npm run pa11y-ci:sitemap > pa11y-ci:sitemap > pa11y-ci --sitemap http://localhost:4000/sitemap.xml --sitemap-exclude "/*.pdf" Running Pa11y on 110 URLs: > http://localhost:4000/guide/glossary - 0 errors > http://localhost:4000/guide/introduction - 0 errors > http://localhost:4000/guide/history - 0 errors > http://localhost:4000/guide/design - 0 errors ... ✔ 110/110 URLs passed

The site passes the tests. Here is an example job running in GitLab. The pa11y configuration continues to test all site pages for accessibility issues and report on them.

What does an error look like? Here is an example:

> http://localhost:4000/guide/introduction - 1 errors Errors in http://localhost:4000/guide/introduction: •

, or elements (https://dequeuniversity.com/rules/axe/3.5/list?application=axeAPI) (#main-content > div:nth-child(2) > div > div > div > div:nth-child(1) > nav > ul)

Mapping the program counter back to the function name in your code

Wed, 02/22/2023 - 16:00

Mapping the program counter back to the function name in your code wcohen Wed, 02/22/2023 - 03:00

Compilers are commonly used to convert human-readable source code into a series of instructions that are directly executed by the computer. A common question is "How do the debuggers and error handlers report the place in the source code where the processor is currently at?" There are various methods to map instructions back to locations in the source code. With the compiler optimizing the code, there are also some complications in mapping the instructions back to the source code. This first article in the series describes how tools map the program counter (also known as the instruction pointer) back to the function name. Subsequent articles in this series will cover mapping the program counter back to the specific line in a source file. It also provides a backtrace describing the series of calls that resulted in the processor being in the current function.

The function entry symbols

When translating source code into an executable binary, the compiler keeps symbol information about the entry to each function. Keeping this information available makes it possible for the linker to assemble multiple object files (.o suffix) into a single executable file. Before the object files are processed by the linker, the final addresses of functions and variables are not known. The object files have placeholders for the symbols that do not yet have addresses. The linker will resolve the addresses when creating the executable. Assuming that the resulting executable is not stripped, those symbols describing the entry of each function are still available. You can see an example of this by compiling the following simple program:

#include #include int a; double b; int main(int argc, char* argv[]) { a = atoi(argv[1]); b = atof(argv[2]); a = a + 1; b = b / 42.0; printf ("a = %d, b = %f\n", a, b); return 0; }

Compile the code:

$ gcc -O2 example.c -o example

You can see where the main function starts at 0x401060 with the following command:

$ nm example |grep main U __libc_start_main@GLIBC_2.34 0000000000401060 T main

Even though the code is compiled without debugging information (-g option), gdb can still find the start of the main function with the symbol information:

$ gdb example GNU gdb (GDB) Fedora 12.1-1.fc36 … (No debugging symbols found in example) (gdb) break main Breakpoint 1 at 0x401060

Step through the code with the GDB nexti command. GDB reports the address it is currently at in the main function:

(gdb) nexti 0x0000000000401061 in main () (gdb) nexti 0x0000000000401065 in main () (gdb) where #0 0x0000000000401065 in main ()

This minimal information is useful but is not ideal. The compiler can optimize functions and split functions into non-contiguous sections to make code associated with the function not obviously tied to the listed function entry. A portion of the function's instructions might be separated from the function entry by other functions' entries. Also, the compiler may generate alternative names that do not directly match the original function names. This makes it more difficult to determine which function in the source code the instructions are associated with.

DWARF information

Code compiled with the -g option includes additional information to map between the source code and the binary executable. By default RPMs with code compiled on Fedora have the debug information generation enabled. Then this information is put into a separate debuginfo RPM, which can be installed as a supplement to the RPMs containing the binaries. This makes it easier to analyze crash dumps and debug programs. With debuginfo, you can get address ranges that map back to particular function names. It also provides the line number and file name that each instruction maps back to. The mapping information is encoded in the DWARF standard.

The DWARF function description

For each function with a function entry there is a DWARF Debug Information Entry (DIE) describing it. This information is in a machine-readable format, but there are a number of tools including llvm-dwarfdump and eu-readelf that produce human-readable output of the DWARF debug information. Below is the llvm-dwarfdump output of the example main function DIE describing the main function of the earlier example.c program.

The DIE starts with the DW_TAG_subprogram to indicate it describes a function. There are other kinds of DWARF tags used to describe other components of programs such as types and variables.

The DIE for the function has multiple attributes each starting with DW_AT_ that describe the characteristics of the function. These attributes provide information about the function, such as where the function is located in the executable binary. It also points where to find it in the original source code.

A few lines down from the DW_TAG_subprogram is the DW_AT_name attribute that describes the source code function name as main. The DW_AT_decl_file and DW_AT_decl_line DWARF attributes describe the file and line number respectively where the function came from. This allows the debugger to find the appropriate location in a file to show you the source code associated with the function. Column information is also included with the DW_AT_decl_column.

The other key pieces of information for mapping between the binary instructions and the source code are the DW_AT_low_pc and DW_AT_high_pc attributes. The use of the DW_AT_low_pc and DW_AT_high_pc indicates that the code for this function is contiguous, ranging from 0x401060 (the same value as provided by nm command earlier) up to but not including 0x4010b7. The DW_AT_ranges attribute is used to describe functions if the function covers non-contiguous regions.

With the program counter, you can map the processor's program counter to the function name and find the file and line number where the function is:

$ llvm-dwarfdump example --name=main example: file format elf64-x86-64 0x00000113: DW_TAG_subprogram DW_AT_external (true) DW_AT_name ("main") DW_AT_decl_file ("/home/wcohen/present/202207youarehere/example.c") DW_AT_decl_line (8) DW_AT_decl_column (0x01) DW_AT_prototyped (true) DW_AT_type (0x00000031 "int") DW_AT_low_pc (0x0000000000401060) DW_AT_high_pc (0x00000000004010b7) DW_AT_frame_base (DW_OP_call_frame_cfa) DW_AT_call_all_calls (true) DW_AT_sibling (0x000001ea)Inlined functions

A compiler can optimize code by replacing a call to another function with instructions that implement the operations of that called function. An inlined function eliminates control flow changes caused by function call and return instructions to implement the traditional function calls. For an inlined function, there is no need for executing additional function prologue and epilogue instructions required to conform to the Application Binary Interface (ABI) of traditional function calls.

Inline functions also provide additional opportunities for optimizations as the compiler can intermix instructions between the caller and the inlined invoked function. This provides a complete picture of what code can safely be eliminated. However, if you just used the address ranges from the real functions described by the DW_TAG_subprogram, then you might misattribute an instruction to the function that called the inlined function, rather than the actual inlined function containing it. For that reason, DWARF has the DW_TAG_inlined_subroutine to provide information about inlined functions.

Surprisingly, even example.c, the simple example provided in this article, has inlined functions in the generated code, atoi and atof. The code block below shows the output of llvm-dwarfdump for atoi. There are two parts, a DW_TAG_inlined_subroutine to describe each place atoi was actually inlined and a DW_TAG_subprogram describing the generic information that does not change between the multiple inlined copies.

The DW_AT_abstract_origin in the DW_TAG_inlined_subroutine points to the associated DW_TAG_subprogram that describes the file with DW_AT_decl_file and DW_AT_decl_line just like a DWARF DIE describing a regular function. In this case, you see that this inlined function is coming from line 362 of system file /usr/include/stdlib.h.

The actual range of addresses that are associated with atof is non-contiguous and described by DW_AT_ranges, [0x401060,0x401060), [0x401061, 0x401065), [0x401068,0x401074), and [0x40107a,0x41080). The DW_TAG_inlined_subroutine has a DW_AT_entry_pc to indicate what location is considered to be the start of the inlined function. With the compiler reordering instructions it may not be obvious what would be considered the first instruction of an inlined function:

$ llvm-dwarfdump example --name=atoi example: file format elf64-x86-64 0x00000159: DW_TAG_inlined_subroutine DW_AT_abstract_origin (0x00000208 "atoi") DW_AT_entry_pc (0x0000000000401060) DW_AT_GNU_entry_view (0x02) DW_AT_ranges (0x0000000c [0x0000000000401060, 0x0000000000401060) [0x0000000000401061, 0x0000000000401065) [0x0000000000401068, 0x0000000000401074) [0x000000000040107a, 0x0000000000401080)) DW_AT_call_file ("/home/wcohen/present/202207youarehere/example.c") DW_AT_call_line (10) DW_AT_call_column (6) DW_AT_sibling (0x00000196) 0x00000208: DW_TAG_subprogram DW_AT_external (true) DW_AT_name ("atoi") DW_AT_decl_file ("/usr/include/stdlib.h") DW_AT_decl_line (362) DW_AT_decl_column (0x01) DW_AT_prototyped (true) DW_AT_type (0x00000031 "int") DW_AT_inline (DW_INL_declared_inlined)

Programming and development Red Hat Developers Blog Programming cheat sheets Try for free: Red Hat Learning Subscription eBook: An introduction to programming with Bash Bash shell scripting cheat sheet eBook: Modernizing Enterprise Java An open source developer's guide to building applications Implications of inline functions

Most programmers think of the processor completely executing one line in the source code before moving on to the next. Similarly, with the expected function call ABI programmers think of where the caller function completes operations in the statements before the call and statements after the call are not started until the callee function return may not hold. With inlined function, the boundaries between functions become fuzzy. Instructions from the caller function may be scheduled before or after the instructions from the inlined functions regardless of where they were in the source code if the compiler determines that the final result will be the same. This may lead to unexpected values when inspecting variables before and after the inlined functions.

Automate OpenStack using Ansible

Tue, 02/21/2023 - 16:00

Automate OpenStack using Ansible ajscanlas Tue, 02/21/2023 - 03:00

I demonstrated how I manage OpenStack using Terraform in my previous article. Using Terraform as Infrastructure as Code (IaC) is a significant advantage, but it also requires the staff to support it. Sometimes, all you really need is the ability to provision infrastructure as timed resources, with no intention of building lasting data structures. When I design a flexible environment, I prefer the declarative language of Ansible.

Ansible's flexibility allows me to design for use cases that include:

A test environment that's timed and can be deleted after use.
A training environment provided for a specific amount of time and then destroyed.
Moving from a manual to an automated infrastructure when Terraform is difficult because it maps your existing infrastructure.

Install Ansible and OpenStack SDK

First, you need to install Ansible and its SDK. For this, you must have Docker or Podman installed.

I use the pip installation of Ansible:

$ python3 -m pip install ansible-navigator --user

The ansible-navigator command gets used by AWX or Red Hat Ansible Automation Platform to run Ansible playbooks. Unlike the previous ansible-playbook or ansible ad-hoc commands, this creates a container and execution environment.

You must build your execution environment with an OpenStack SDK. For this, I strongly recommend the article by Gineesh Madapparambath on how to create an execution environment.

After creation, use your Ansible execution environment to configure your image:

$ ansible-navigator images

Choose which image to use and run a playbook:

$ ansible-navigator run main.yaml --stdout

More on Ansible 5 reasons to migrate to Red Hat Ansible Automation Platform 2 A quickstart guide to Ansible Ansible cheat sheet Free online course: Ansible essentials Download and install Ansible eBook: The automated enterprise eBook: Ansible for DevOps Free Ansible eBooks Latest Ansible articles Sample playbook

My sample playbook uses the same structure I demonstrated in my Terraform article:

$ cat >> main.yaml << EOF --- - hosts: localhost vars: flavors: - name: "small" ram: 4096 vcpus: 1 - name: "medium" ram: 8096 vcpus: 2 - name: "large" ram: 16384 vcpus: 4 - name: "xlarge" ram: 32768 vcpus: 8 tasks: - name: create flavors openstack.cloud.compute_flavor: state: present name: "{{ item.name }}" ram: "{{ item.ram }}" vcpus: "{{ item.vcpus }}" disk: 0 loop: - flavors - name: create external network openstack.cloud.network: state: present name: "external-network" provider_network_type: "flat" provider_physical_network: "physnet1" external: true - name: create external subnet openstack.cloud.subnet: state: present name: "external-subnet" network_name: "external-network" cidr: "10.0.0.0/8" gateway_ip: "10.0.0.1" dns_nameservers: - "10.0.0.254" - "10.0.0.253" allocation_pool_start: "10.0.0.2" allocation_pool_end: "10.0.254.254" - name: create external router openstack.cloud.router: name: "external-router" state: present network: "external-network" - name: create Cirros image openstack.cloud.image: name: cirros container_format: bare disk_format: qcow2 state: present filename: cirros-0.6.1-x86_64-disk.img - name: create Demo project openstack.cloud.project: state: present name: "Demo" enabled: True - name: create demo-user openstack.cloud.user: state: present name: "demo-user" password: "demo" default_project: "Demo" EOFResults

After you run the playbook, you get networks:

Image by:

(AJ Canlas, CC BY-SA 4.0)

You also get flavors:

Image by:

(AJ Canlas, CC BY-SA 4.0)

And images:

Image by:

(AJ Canlas, CC BY-SA 4.0)

And a demo project:

Image by:

(AJ Canlas, CC BY-SA 4.0)

The right tool

There's no single right or wrong answer for how you automate. How you implement your automation depends on your organization's needs, infrastructure, system design, and staff resources. Select one of the great open source tools out there to get started. Put in the effort now, and save yourself a lot of work in the future.

Ansible offers unique flexibility when automating your infrastructure.

Image by:

opensource.com

OpenStack Ansible Cloud Automation What to read next How to use GitOps to automate Terraform This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

My favorite open source tools for personal finance

Tue, 02/21/2023 - 16:00

My favorite open source tools for personal finance Don Watkins Tue, 02/21/2023 - 03:00

There are tons of open source personal finance apps out there. These apps are useful because they help you take control of your finances. I find it much easier to use an app rather than a spreadsheet or a ledger. Here are some of my favorite apps.

GnuCash

GnuCash is a popular open source personal finance or small business accounting program for Linux users. It is a full-featured double-entry accounting system that supports multiple accounts, investments, and currency conversions. The app also provides a variety of helpful financial tools, including budgeting, reporting, invoicing, and more. GnuCash is a great choice for users looking for a comprehensive and powerful financial management solution. GnuCash is my own personal favorite that I have been using every day for over six years. GnuCash can be installed from the command line or as a Flatpak depending on your distribution and personal choice. GnuCash is licensed with the Gnu Public License. GnuCash is also available for MacOS and Windows users too.

Homebank

Hom ebank is a free, open source personal finance app for Linux, Windows, and MacOS users. It easily imports files from Quicken, Microsoft Money, and other software. It also imports from popular bank formats OFX/QFX, QIF, and CSV. The source code for Homebank is freely available and licensed with GPL v 2.0. The project provides information about how to download and install it on your distribution or operating system.

Our favorite resources about open source Git cheat sheet Advanced Linux commands cheat sheet Open source alternatives Free online course: RHEL technical overview Check out more cheat sheets KMyMoney

KMyMoney is a cross-platform double-entry bookkeeping system for personal finance management built on KDE. It's similar in operation to popular proprietary personal finance applications. KMyMoney also supports multiple accounts and currencies, making it a great choice for users who need to manage their finances in multiple countries. Its latest stable release was in June 2021. It is licensed with the Gnu Public License. The project provides download images for Linux, Windows, and MacOS.

Skrooge

Skrooge is open source and available for download and installation on Linux and BSD. It's also available on Windows and MacOS. You can import accounts from many sources including AFB120, QIF, CSV, MT940, OFX, and QFX. According to their website Skrooge can import transactions directly from all your banks' websites in one click. Skrooge provides excellent documentation, too.

These are just a few of the applications out there that can help you take control of your finances. Try one out or try them all out. See which one works best for you. To all my friends in the United States, don't forget to be ready for tax season!

Try one of these open source apps to keep your budget in check.

Image by:

Opensource.com

Tools Business What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

4 questions open source engineers should ask to mitigate risk at scale

Mon, 02/20/2023 - 16:00

4 questions open source engineers should ask to mitigate risk at scale kathryn.xtang@… Mon, 02/20/2023 - 03:00

At Shopify, we use and maintain a lot of open source projects, and every year we prepare for Black Friday Cyber Monday (BFCM) and other high-traffic events to make sure our merchants can sell to their buyers. To do this, we built an infrastructure platform at a large scale that is highly complex, interconnected, globally distributed, requiring thoughtful technology investments from a network of teams. We’re changing how the internet works, where no single person can oversee the full design and detail at our scale.

Over BFCM 2022, we served 75.98M requests per minute to our commerce platform at peak. That’s 1.27M requests per second. Working at this massive scale in a complex and interdependent system, it would be impossible to identify and mitigate every possible risk. This article breaks down a high-level risk mitigation process into four questions that can be applied to nearly any scenario to help you make the best use of your time and resources available.

1. What are the risks?

To inform mitigation decisions, you must first understand the current state of affairs. We expand our breadth of knowledge by learning from people from all corners of the platform. We run “what could go wrong” (WCGW) exercises where anyone building or interested in infrastructure can highlight a risk. These can be technology risks, operational risks, or something else. Having this unfiltered list is a great way to get a broad understanding of what could happen.

The goal here is visibility.

2. What is worth mitigating?

Great brainstorming leaves us with a large and daunting list of risks. With limited time to fix things, the key is to prioritize what is most important to our business. To do this, we vote on risks, then gather technical experts to discuss highest ranked risks in more detail, including their likelihood and severity. We make decisions about what and how to mitigate, and which team will own each action item.

The goal here is to optimize how we spend our time.

3. Who makes what decisions?

In any organization, there are times when waiting for a perfect consensus is not possible or not effective. Shopify moves tremendously fast because we make sure to identify decision makers, then empower them to gather input, weigh risks/rewards, and come to a decision. Often the decision is best made by the subject matter expert or who bears the most benefit or repercussions of whatever direction we choose.

The goal here is to align incentives and accountability.

Our favorite resources about open source Git cheat sheet Advanced Linux commands cheat sheet Open source alternatives Free online course: RHEL technical overview Check out more cheat sheets 4. How do you communicate?

We move fast but still need to keep stakeholders and close collaborators informed. We summarize key findings and risks from our WCGW exercises so that we all land on the same page about our risk profile. This may include key risks or single points of failure. We over-communicate so that we’re aligned and aware and stakeholders have opportunities to interject.

The goal here is alignment and awareness.

Solving the right things when there is uncertainty

Underlying all these questions is the uncertainty in our working environment. You never have all the facts or know exactly which components will fail when and how. The best way to deal with uncertainty is by using probability.

Expert poker players know that great bets don’t always yield great outcomes, and bad bets don’t always yield bad outcomes. What’s important is to bet on the probability of outcomes, where over enough rounds, your results will converge to expectation. The same applies in engineering, where we constantly make bets and learn from them. Great bets require clearly distinguishing the quality of your decisions versus outcomes. It means not over-indexing on bad decisions that led to lucky outcomes or great decisions that happen to run into very unlucky scenarios.

Knowing that we can’t control everything also helps us stay calm, which is vital for us to practice good judgment in high-pressure situations.

When it comes to BFCM (and life in general), no one can predict the future or fully protect against all risks. The question is, what would you change looking back? In hindsight, would you feel confident that you prioritized the most important things and made thoughtful bets using the information available? Did you facilitate meaningful discussions with the right people? Could you justify your actions to your customers and their customers?

This article originally appeared on Planning in Bets: Risk Mitigation at Scale and is republished with permission.

What do you do with a finite amount of time to deal with an infinite number of things that can go wrong?

Image by:

Opensource.com

Business DevOps SCaLE What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Kubernetes policy engines: OPA vs. Kyverno vs. jsPolicy

Mon, 02/20/2023 - 16:00

Kubernetes policy engines: OPA vs. Kyverno vs. jsPolicy JosephEshiett Mon, 02/20/2023 - 03:00

A Kubernetes policy engine is essential for keeping your cluster safe and ensuring policies are set correctly at the outset. For example, you probably need a policy to control who has the authority to set a privileged pod. These engines define what end users can do on the cluster and ensure that clusters can communicate. Any time a Kubernetes object is created, a policy evaluates and validates or mutates the request. Policies can apply across a namespace or different pods with a specific label in the cluster.

Kubernetes policy engines block objects that could harm or affect the cluster if they don't meet the policy's requirements. Using policies enables users to build complex configurations that other tools, such as Terraform or Ansible, cannot achieve.

The policy landscape has evolved in recent years, and the number of policy engines available continues to increase. Newer products compete against well-established tools.

This article highlights some features you should look for in a policy engine and where these three examples excel and underperform. It compares three popular open source policy engines, Open Policy Agent (OPA), Kyverno, and jsPolicy.

Policy engine features

I'll begin by listing various features so you can compare the policy engines:

Supported language: A policy engine must use a language supported by Kubernetes for easy management of policy resources.
Validation: Validation rules decide the properties with which a resource can be created. Resources are validated when they are checked against the rules and accepted.
Mutation: Mutation rules can modify specific resources in the cluster. These rules modify a particular object in a given way.
Tooling for development and testing: These tools test a set of resources against one or more policies to compare the resources against your desired results (declared in a separate file).
Package management: Package management handles where your rules are stored and how they are managed in the cluster.
Image verification: The use of policies to verify and sign container images.
Extensions: Custom-built functions and plugins that extend and implement functionality, like support for new protocols.
Metrics: Monitoring any applied changes to policies, activities related to incoming requests, and the results produced as an outcome.

Open Policy Agent (OPA)

Open Policy Agent (OPA) is an easy-to-use policy engine that can be colocated with your service and incorporated as a sidecar, host-level daemon, or library. OPA is a general-purpose engine that manages policies across several stacks, and you can utilize it for other tasks like data filtering and CI/CD pipelines.

OPA allows you to decouple your policies from your infrastructure, service, or application so that people responsible for policy management can control the policy separately from the service. You can also decouple policies from any software service you like and write content-aware policies using any context you want. Decoupling policies will help you build services at scale, improve the capacity to locate violations and conflicts, and reduce the risk of human errors.

OPA policies use a language called Rego. Rego is a query language that extends Datalog to support structured data models such as JSON. OPA provides a framework to write tests for your policies. This framework speeds up the development of new rules and reduces the time to modify existing ones. OPA can also report performance metrics at runtime. These metrics can be requested on individual API calls and are returned in line with the API response.

OPA works by making decisions on policies, not necessarily enforcing them. When a query is sent into the system, it gets passed to OPA, which then validates the query against the policies in place and makes a decision. OPA makes policy decisions by comparing the query input against policies and data. OPA and Rego are domain-agnostic, meaning you can describe any invariant in the policies. Also, policy decisions are not restricted to yes/no or allow/deny answers. Like query inputs, your policies can create structured data as an output.

Kyverno

Kyverno is a Kubernetes policy engine that employs a declarative management paradigm to construct policies for changing, validating, or generating resources or configurations. In contrast to OPA, which uses policy as code, you specify the code rather than write it, and Kyverno then figures out how to execute it. Kyverno uses YAML, so these policies are taken as Kubernetes resources and can be written without learning a new language. This makes it easy to view and process policy results. Kyverno outshines OPA here, as developing code in the Rego language can be difficult, especially without in-depth knowledge.

Kyverno works well with other developer tools like Git and kubectl. Validation rules are the primary use case for admission controllers like Kyverno, which makes it very easy to validate resources that respect the policy rules when creating them. Kyverno uses Cosign to verify and sign images. If the image is not found in the OCI registry or was not signed using the specified key, the policy rule will not validate it. Also, Kyverno uses Grafana to expose and collect metrics from the cluster. It simplifies and consolidates policy distribution using a container registry (OCI registry).

Kyverno works as a dynamic admission controller, receiving HTTP callbacks from the Kubernetes API server and applying matching policies to these callbacks. The policies match the resources using selectors like name, kind, and label. Kyverno uses a webhook as the controller as it handles admission review requests from the server. In the webhook, a monitor creates and manages all the required configurations. There's a generator controller that generates requests and manages the span of generated resources, and a policy controller that creates, updates, deletes, and watches policy resources, running background scans at intervals to decide what course of action to take.

More on Kubernetes What is Kubernetes? Free online course: Containers, Kubernetes and Red Hat OpenShift technical over… Test drive OpenShift hands-on An introduction to enterprise Kubernetes How to explain Kubernetes in plain terms eBook: Running Kubernetes on your Raspberry Pi homelab Kubernetes cheat sheet eBook: A guide to Kubernetes for SREs and sysadmins Latest Kubernetes articles jsPolicy

jsPolicy is an open source policy engine for Kubernetes that lets users build policies using JavaScript or TypeScript. Managing policies with JavaScript is less complex and more straightforward. Due to the widespread use of the JavaScript programming language, frameworks, and numerous libraries and modules, jsPolicy is a natural choice as a policy engine tool. Kyverno and OPA are more difficult to alter and validate than jsPolicy. Its distribution uses npm packages and features a built-in JavaScript SDK for creating and packaging policies.

jsPolicy is the first policy engine to have controller policies (policies that respond to Kubernetes events). This feature lets you do something on Kubernetes events and validate or mutate them using jsPolicy. Like OPA, jsPolicy is a policy-as-code platform. However, it was created to solve the Rego language problem and provide some functionality for some features not available on Kyverno.

You can use kubectl and the Kubernetes API with jsPolicy, and every request that comes in persists in etcd. Before the requests get persisted, a webhook manager will execute the policy inside your cluster. It uses prebuilt JavaScript sandboxes in your cluster to aid policy execution, increasing efficiency and speed. A policy compiler reads the jsPolicy code and compiles it into a policy bundle, which is placed and run in the sandbox to provide policy violations. The policy violation is queried for alerting and auditing objects that violate the policy code. Since jsPolicy lets you work with JavaScript, you can use the entire JavaScript ecosystem with its great dev tools and frameworks for testing to write, test, and maintain policies.

Compare Kubernetes policy engines

The features of each policy engine differ. While they can all validate and mutate resources, they differ in other specific functions. For instance, OPA and Kyverno support extensions, but jsPolicy does not. The summary below compares the features of these three policies:

Image by:

(Joseph Eshiett, CC BY-SA 4.0)

OPA
- Language: Rego
- Validation: Yes
- Mutation: Alpha
- Development/testing: Limited
- Package management: NA
- Image validation: Yes
- Extensions: Yes
- Metrics: Prometheus
Kyverno
- Language: YAML
- Validation: Yes
- Mutation: Yes
- Development/testing: Limited
- Package management: NA
- Image validation: Yes
- Extensions: Yes
- Metrics: Grafana
jsPolicy
- Language: JavaScript
- Validation: Yes
- Mutation: Yes
- Development/testing: Extensive
- Package management: npm
- Image validation: No
- Extensions: No
- Metrics: Prometheus

Wrap up

This article discussed the concepts surrounding Kubernetes policy engines and compared three different Kubernetes policy engines: OPA, Kyverno, and jsPolicy.

Deciding which engine to use depends on your personal preference. If you'd like a more direct and simple approach, or if you're well-versed in JavaScript and TypeScript, you should use jsPolicy. But, if you prefer YAML and want to stick with working directly with Kubernetes resources, Kyverno is a good option, too.

Learn what features to look for in a Kubernetes policy engine.

Image by:

Opensource.com

Kubernetes What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

What does ChatGPT mean for the open source community?

Sat, 02/18/2023 - 16:00

What does ChatGPT mean for the open source community? sethkenlon Sat, 02/18/2023 - 03:00

Machine learning and "artificial intelligence" is on a lot of people's minds right now, largely because of ChatGPT. With its widely-available public demonstration, the not-so-aptly named OpenAI group (ChatGPT is not open source) has shown the public that when you point a whole cloud of computing power back at the internet, you can generate believable text about nearly any subject. As many people have pointed out, there's a big difference between "believable" and "correct" of course, but on a superficial level it seems like ChatGPT is a valid source of surface-level summary output. ChatGPT isn't open source, but nearly everything it outputs is based on open knowledge. It's based on content you and I have put onto the internet for others. Does this mean ChatGPT has joined a community? Is ChatGPT contributing to improving shared knowledge? Or does it just reduce how many internet searches you have to do before arriving at a general idea of what might be an answer to your question?

Benefits of the contrary

You're probably a member of some community, whether it's a community of an open source project or even your local neighborhood. Either way, you've probably noticed that sometimes people can be annoying. It's a fact of life that people have opinions, and often those opinions are in conflict with one another. When there's a disagreement over how something ought to be done, it usually feels like time's being wasted. After all, you know the best solution, but instead of putting it into action, you have to spend all day convincing everyone else of its merit. It would be so much easier if everyone would just agree with you, right?

Disagreement is also uncomfortable. It leads to difficult conversations. You have to find a compromise or else convince somebody to see things your way, even as they try to convince you to see things their way. It's not easy, and it's often not what you want to be doing at any given time.

Of course, most adults understand that there's power in the contrary. A bot might be able to emulate a contrary opinion, but there's a difference between an opinion and stubbornness or obstinacy. Differing opinions, formed from expertise and experience, are vital for successful and fruitful collaboration. As uncomfortable as they may be, differing opinions on the "right" way to do something is the best way to stress test your ideas. By looking at the contrary, you can identify your preconceptions, biases, and assumptions. By accepting differing opinions, you can refine your own.

Spark of originality

A bot armed with machine learning can only invent ideas from existing ideas. While there may be value in distilling noise into something singularly tangible, it's still just a summary of notions that have come before. A gathering of real human actual minds is powerful because of the seemingly irrelevant and unexpected ideas that form from conversation, iteration, agreement, disagreement, and diversity of experiences and backgrounds. It might not make logical sense for me to base my CI/CD pipeline on the strategy I invented for last night's tabletop roleplaying game, but if that served as inspiration for something that ends up being really good then it doesn't matter in the end. There's an irrationality to interpreting the world through your experience embroidering or gardening or cooking or building LEGO sets with your kid, but that doesn't make it invalid. In fact, it's the ability to connect inspiration to action that gives birth to invention. That's not something ChatGPT can learn from the internet.

AI and machine learning with open source What is AI/ML? Cheat sheet: AI glossary eBook: A practical guide to home automation using open source tools Why deploy AI/ML workloads on OpenShift Develop, train, and test your ML workloads with Red Hat OpenShift Data Science Open Source Stories: Road to AI What is edge machine learning? Latest AI and machine learning articles System design

ChatGPT and other AI experiments may well have their use in reducing repetitious tasks, or for catching potential bugs, or for getting you started with a particularly confounding YAML file. But maybe the hidden message here is actually a question: why do we think we need ChatGPT for these things? Could it be that, after all, these processes need improvement themselves? Could it be that maybe writing some "simple" YAML isn't as simple as it at first seemed? Maybe those bugs that need an artificial intelligence to catch are less a disease than a symptom of over-complex language design or a failure in how we teach code, or just an opportunity to develop easier entries into programming.

In other words, maybe machine learning bots aren't the solution to anything, but an indication of where we're doing a disservice to ourselves. In open source, we design the systems we interact with. We don't have to design chat bots to help us understand how the code works or how to program, because we're the inventors. We can redesign around the problems. We don't need a chat bot to coalesce and condense the confusion of the worldwide community, because we can create the best solution possible.

Human connection

Community is about people. Making connections with other people with a shared interest and passion for something is what makes communities so fulfilling. Both the disagreements and the moments of shared inspiration are profound experiences that we humans bring to one another in our forums, chat rooms, bug reports, conferences, and neighborhoods. As an open source community, we create technology. We create it openly, together, and with a genuine interest in sharing experiential knowledge. We value diversity, and we find value in the perspectives of novices and experts alike. These are things you can't distill in machine learning chat bot, whether it's open source or not (and ChatGPT is not).

The open source community thrives on the genuine interest in sharing. That's something ChatGPT cannot emulate.

AI and machine learning Community management What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 3 Comments Register or Login to post a comment. Don Watkins | February 18, 2023 Register or Login to like

Great insights Seth! You've helped me to sort this out. I'm fascinated by ChatGPT and other AI instances. However, there is a lack of human connection and insight. Definitely no community and it occurs to me that we only get responses on the data that's fed into these engines. Helpful in some instances but without context and certainly minor voices are never heard.

sushidub | February 21, 2023 Register or Login to like

Love this entire perspective on ML and AI lacking the human component. Let's face it - AI is not human, nor will it ever be. That is, of course, unless I myself happen to be a bot programmed to be unaware of the fact that it is one. At the moment I see and hear lot of grandiose speculation going around on the superpower of ML and AI - most of which seems less anecdotal than it is coming from a trickle of reporting regarding the ChatGPT service and the experience a handful of individuals have had thus far. The tube has great reach these days. The implications coming from the chatter far outweigh the actual capability of the OpenAI service and unsurprisingly are very human in both pessimism and optimism regarding its potential. I won't deny the awesomeness and power of ML in its current and potential state, but for now I see most of the compute spend heading in an automation direction (mundane taskery) and actually surprised at the amount of hype increase when considering competent ML procedural work has been around for almost a decade now, if not longer.

Craig McClanahan | February 22, 2023 Register or Login to like

An issue I have not seen covered much is important to many open source communities: licensing.

Many orgs will not accept even human originated contributions that were copied from another project with an incompatible license. What are they supposed to do with an AI generated code snippet from an AI entity trained on “open” code that may be publicly visible, but sourced under such an incompatible license?

Stargaze from the web browser with an open source planetarium

Fri, 02/17/2023 - 16:00

Stargaze from the web browser with an open source planetarium cherrybomb Fri, 02/17/2023 - 03:00

If you've seen the news recently, you'll have noticed many articles on the comet with the green tail (C/2022 E3) that will only come once in a lifetime for everyone to see.

In theory, you can see the comet through a telescope across several days, but I don't own a telescope. So my window to see the comet was down to one night when it would be visible without any special equipment.

As of writing this article, that would be tonight.

But as I write this article, there's full cloud coverage and rain.

So there's no comet sighting for me, at least not in real life. I looked up some pictures. I discovered Stellarium, an amazing open source planetarium for your computer! This article is all about this open source way to view the universe.

Stellarium

Stellarium is a free, open source planetarium software for every OS. It's an amazing 3D simulation of the night sky that lets you explore and observe stars, constellations, planets, and other celestial objects. What's really cool about it is that it has a comprehensive database of stars, nebulae, galaxies, and so much more. Also, the software allows you to view the sky from any location on Earth at any time in the past or future, so you can look at a future night sky your descendants may see. Stellarium also includes advanced features such as telescopic controls and the ability to track the motion of celestial objects. Amateur astronomers and educators use Stellarium to explore the sky, and it is considered one of the best planetarium programs available.

What does Stellarium offer?

This is the coolest part. With the most accurate database, you get some super high-quality maps in the most realistic way possible. You can create customized views and pick a time and location of what you want to see. If you think, "It really can't be that realistic," you would be wrong. It also simulates the atmosphere. Your view will look as if you were standing on the ground staring at the sky yourself. There's an option to act like you're looking through a telescope which is just as real as you doing it yourself from your backyard. It also includes a scripting language that allows you to create animated events—great if you're teaching or just want to mess around. Finally, it's available in a wide range of languages, meaning almost anyone can use it.

Do I have to download and install this to use it?

Nope, you don't, and I'll give you a bit of a demo on how to use the web version.

First, get to the website. It will show you the sky. It asks for your location on your browser window. From where I was at this point of writing, I lucked out and caught the International Space Station (ISS) hanging out on my screen. It didn't last long, so I'm glad I grabbed it while I did.

Image by:

(Jess Cherry, CC BY-SA 4.0)

I'll explain those buttons from right to left.

First on the list is the Full Screen button. If you want to sit and stare into space (pun intended), click that square, and you're full screen ahoy!

Next is a Night Mode button. Use this in the dark so your night vision won't be interfered with. I wasn't using this at night, so it looks rather odd.

Image by:

(Jess Cherry, CC BY-SA 4.0)

The Deep Sky Objects button is next, which is what those tiny circles are on the screen. When you click within a circle, it shows detailed information about that object. It also includes a link to additional resources if you want to research further.

Image by:

(Jess Cherry, CC BY-SA 4.0)

Additional deep sky details.

Image by:

(Jess Cherry, CC BY-SA 4.0)

There is the Equatorial Grid button. This coordinate system is based on two measurements similar to latitude and longitude but made from right ascension and declination. Right ascension is measured in hours, minutes, and seconds eastward along the celestial equator (which is equal to 24 hours) and is actually the known equator on every globe you see. I could go into further detail, but I suggest checking out resources from your local college or NASA. Here's what happens when you click that button.

Select the free grid, which centers right on Polaris, more commonly known as the North Star.

Image by:

(Jess Cherry, CC BY-SA 4.0)

Additional details are shown.

Image by:

(Jess Cherry, CC BY-SA 4.0)

The next button over is the Azimuthal Grid. This grid type is based on altitude and azimuth relative to your location. Azimuth is an angular measurement in decimal degrees. Again, check with your local planetarium, college, educators, and NASA for more details.

Image by:

(Jess Cherry, CC BY-SA 4.0)

This picture shows the location I'm technically "standing" and gives a good view.

Image by:

(Jess Cherry, CC BY-SA 4.0)

Next is the Landscape button. When you click it, the scenic landscape disappears.

Image by:

(Jess Cherry, CC BY-SA 4.0)

I liked the farm better, so I intend to turn it back on.

Remember I explained there's a realistic view of how the atmosphere visually interferes? Well, it's been on the entire time in my screenshots. You've seen the refractional light providing a filter like you would any other night. The Atmosphere button removes that. I turned off that visual in the following screenshot to show you a whole new world.

Image by:

(Jess Cherry, CC BY-SA 4.0)

And as you can see, a whole new level of the night sky opens up. While in theory, this would be amazing to see, we need oxygen, so the atmosphere needs to stay. However, I'll leave this button off for the remaining images in the tour.

Another entertaining button (to me, at least) is the Constellation Art button. You may be thinking, "Ok, cool, we get to see those lines in the sky like every other map I've seen." Nope! You get to see some pretty amazing artwork based on all those historical myths you've read about.

Image by:

(Jess Cherry, CC BY-SA 4.0)

This is a fantastic function, and I love the detail. However, the Constellations button displays the traditional constellation lines if you prefer that view. Aside from the lines, it also includes names. When you click those names, you get information and the picture shown in the above artwork.

Image by:

(Jess Cherry, CC BY-SA 4.0)

Image by:

(Jess Cherry, CC BY-SA 4.0)

What else can I do?

So now that I introduced the buttons, I'll show the other features, starting with movement on the screen. It's a simple click and drag to change your view of the sky. When you remove the landscape, you can look down and see everything below you. There's a light-colored fog circle around your view, which is the celestial equator.

Earlier, I mentioned some cool telescope-related features. You can easily manage these. Start by clicking an object, star, or planet and using a few buttons. I'll click the Moon. Underneath the description, you'll see what looks like a target with a star in the middle.

Image by:

(Jess Cherry, CC BY-SA 4.0)

Click that target icon to get a set of buttons to zoom in and out.

Image by:

(Jess Cherry, CC BY-SA 4.0)

I zoomed in with the plus (+) button to see what I would get. The picture below appeared after I clicked the plus button about five times.

Image by:

(Jess Cherry, CC BY-SA 4.0)

Open science and sustainability Video series: ChRIS (ChRIS Research Integration System) Explore Red Hat Research projects 6 articles to inspire open source sustainability How Linux rescues slow computers (and the planet) Latest articles about open science Latest articles about open education Latest articles about sustainability

There's also a link button next to those zoom buttons allowing me to share this view at the exact time and date I wrote this article.

How do I contribute?

Since this is open source software, you can contribute in whatever way you can. See these contributing guidelines, ask to commit code via pull request (PR), report problems, or become a backer or sponsor of the project by donating here.

Space out with Stellarium

I stumbled on this amazing project while wandering the internet in search of something. I mentioned Stellarium is installable on every OS. In a future article, I'll cover installing and using it on a Raspberry Pi. I'm thinking of titling it, "The galaxy in your hands," but I'm always open to title suggestions. Feel free to leave a comment with anything else you want to read about. Hopefully, you learned something new and will enjoy the web version of Stellarium. And maybe you just "spaced out" at work for a few minutes.

Stellarium is an open source planetarium that enables you to explore the night sky right on your own computer.

Image by:

Greg Rakozy via Unsplash

Science What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 3 Comments Register or Login to post a comment. Don Watkins | February 17, 2023 Register or Login to like

Wow! This is great! I've known about Stellarium and even written about it before but I didn't realize that they had it on the web. That is so cool. Thank you for writing this and providing all the "how-to" information. I sent a link to our children and grandchildren who will be fascinated.

pdecker | February 17, 2023 Register or Login to like

I've been using it for years. Like you I didn't have a very good chance to see the comet in my area. There were clear nights, but too cold to spend much time searching. All I have are binoculars. Maybe I'll catch the next one.

I've used the web a little bit, but the application on Windows and Linux much more. I tried to install on my Android phone in the past and had a bit of a problem at that time. I think it was a purchased app with a demo version maybe. I don't remember right now.

One of the neatest things you can do with the application is to speed up, slow down, and pause time to see exactly what you're looking for. This can be super helpful to plan a viewing session. A lunar eclipse looks really good too.

Alan Formy-Duval | February 20, 2023 Register or Login to like

Very cool! I will have to give this a try very soon!

A guide to SLEs and SLAs for open source projects

Thu, 02/16/2023 - 16:00

A guide to SLEs and SLAs for open source projects quaid Thu, 02/16/2023 - 03:00

The term Service Level Agreement (SLA) is a familiar one, particularly in the context of a cloud or managed service on the web. An SLA refers to the contractual obligations a service provider has to its customers and is the instrument defining permissible performance levels for the service. For example, a service agreement might determine a service level of 99.95% uptime, with penalties for falling under 99.95% uptime (more than about 4.5 hours of downtime in a year or 1.125 hours per quarter).

The term is so useful for describing both requirements and expectations around service uptime that it has been co-opted for other uses where a contractual agreement doesn't or can't exist. For example, a community SLA or free-tier SLA might describe a non-contractual situation with the desire or expectation of maintaining a certain service level.

The problem with this usage is a wonky but important one. In an SLA, "agreement" always means a contract; the contextual meaning of the word cannot be translated to other contexts. The relationship between two or more people is, by nature, non-contractual. That's why contracts were invented: to provide a way to formalize an agreement and its terms beyond the moment of coming to an agreement.

Misusing the term SLA creates specific problems in at least two areas:

In cloud-native site/system reliability engineering (SRE), two of the tools central to the practice are the Service Level Objectives (SLO), created to make sure user experiences are within an acceptable range, and the Service Level Indicator (SLI) used to track the status and trends of the SLO. Both of these roll up to an SLA in a commercial situation, but there's no good equivalent to roll up to in a non-commercial situation.
In some cases, managed cloud services are delivered to a user base, but there isn't a contractual dynamic, for example, with IT services in academic settings and open source services delivered as part of an open source project. The groups need a way to frame and discuss service levels without a contractual element.

This bit of word-wonkiness and nerdery is important to my work on the Operate First project, because part of our work is creating the first all open source SRE practice. This includes not only having SLOs/SLIs but also documenting how to write them. We do this because Operate First is an upstream open source project where the content will likely be adopted for use in a commercial context with an SLA.

As the community architect for the Operate First project, I am advocating for adopting the similar, well-used term Service Level Expectation (SLE) as the top-level object that we roll Service Level Objectives (SLOs) up to. This term reflects the nature of open source communities. An open source community does not produce its work due to a contractual agreement between community members. Rather, the community is held together by mutual interest and shared expectations around getting work done.

Put another way, if a team in an open source project does not finish a component that another team relies on, there is no SLA stating that Team A owes monetary compensation to Team B. The same is true for services operated by an open source project: No one expects an SLA-bound, commercial level of service. Community members and the wider user base expect teams to clearly articulate what they can and cannot do and generally stick to that.

I will share my proposal that a set of SLOs can be constructed to remain intact when moving from an SLE environment to an SLA environment. In other words, the carefully constructed SLIs that underlie the SLOs would remain intact going from a community cloud to a commercial cloud.

But first, some additional background about the origin and use of SLEs.

SLEs in the real world

Two common places where SLEs are implemented are in university/research environments and as part of a Kanban workflow. The concluding section below contains a list of example organizations using remarkably similar SLEs, including institutions like the University of Michigan, Washington University in St. Louis, and others. In a Kanban workflow, an SLE defines the expectations between teams when there are dependencies on each other's work. When one team needs another team to complete its work by a certain deadline or respond to a request within a specific time period, they can use an SLE that is added to the Kanban logic.

In these situations, there may be time and response information provided or understood from a related context. Staff sysadmins might be on duty in two shifts from 8AM to 8PM, for example, five days a week. The published expectation would be 5x12 for non-critical issues, with some other expectation in place for the critical, all-services-and-network-disrupted type of outages.

In an open source project, developers may be balancing time working on developing their product with supporting the product services. A team might offer to clear the issue and bug queue after lunch Monday through Thursday. So the SLE would be 4x4 for non-critical situations.

Our favorite resources about open source Git cheat sheet Advanced Linux commands cheat sheet Open source alternatives Free online course: RHEL technical overview Check out more cheat sheets What are cold-swappable SLOs?

The core idea here is to design a set of SLOs that can be moved from under an SLE to an SLA without changing anything else.

An SLE has a focus of expectation, which can be thought of generally as ranging from low-expectation to high-expectation environments. Thus, the act of writing an SLO/SLI combo to work with an SLE environment helps to document the knowledge of how to range the measurement on the indicator for this service depending on how it's used, setup, and so on.

Establish an SLE with details for different services (if they have different uptime goals) and clarify boundaries, such as, "Developer teams respond to outages during an established window of time during the work week."
Developers and operators establish one to three SLOs for a service, for example, "Uptime with 5x5 response time for trouble tickets," meaning Monday-Friday from 12:00 to 17:00 UTC (5x5).
SLIs are created to track the objective. When writing the spec for the SLI, write for the specific and the generic case as much as possible. The goal is to give the reader a high percentage of what they need to implement the pattern in their environment with this software.

8 examples of SLEs

Although not in universal usage, I found many examples of SLEs in academic and research settings, an open source community example (Fedora and CentOS communities), and a very similar concept in Kanban of the expectations for seeing a sprint through from start to finish.

I'll conclude this article with a non-exhaustive list of the introductory content from each page:

University of Michigan ITS general SLEs:

The general campus Service Level Expectation (SLE) sets customer expectations for how one receives ITS services. The SLE reflects the way Information and Technology Services (ITS) does business today. This SLE describes response times for incidents and requests, prioritization of work, and the outage notification process.

Specific services may have additional levels of commitment and will be defined separately under a service-based SLE.

Washington University in St. Louis (2016) SLEs for basic IT services for all customers:

This document represents the Service Level Expectation (SLE) for the Washington University Information Technology (WashU IT) Basic Information Technology (BIT) Bundle Service.

The purpose of this agreement is to ensure that this service meets customer expectations and to define the roles/responsibilities of each party. The SLE outlines the following:

Service Overview
Service Features (included & excluded)
Service Warranty
Service Roles & Responsibilities
Service Reporting & Metrics
Service Review, Bundles & Pricing

Each section provides service and support details specific to the BIT Bundle Service as well as outlining WashU IT's general support model for all services and systems.

Rutgers (2019) SLE for virtual infrastructure hosting:

Thank you for partnering with us to help deliver IT services to the university community. This document is intended to set expectations about the service Enterprise Infrastructure Systems Engineering delivers as well as how to handle exceptions to that service.

Western Michigan University SLEs:

This Service Level Expectation document is intended to define the following:

A high-level description of services provided by the Technology Help Desk.
The responsibilities of the Technology Help Desk.
When and how to contact the Technology Help Desk.
The incident/work order process and guidelines.

The content of this document is subject to modifications in response to changes in technology services/support needs and will remain in effect until revised or terminated.

University of Waterloo SLEs for core services:

The purpose of this document is to define the services applicable, and provide other information, either directly, or as references to public web pages or other documents, as are required for the effective interpretation and implementation of these service level expectations.

University of Florida Research Computing SLEs:

This page describes the service level expectations that researchers should keep in mind when storing data and working on the HiPerGator system.

There are three categories of service to be considered. Please read these service descriptions carefully.

The Fedora and CentOS Community Platform Engineering (CPE) SLEs for community services:

The CPE team does not have any formal agreement or contract regarding the availability of its different services. However, we do try our best to keep services running, and as a result, you can have some expectations as to what we will do to this extent.

Kanban:

SLEs can be defined as forecasts of cycle time targets for when a given service should be delivered to a customer (internal or external)...

Service Level Expectations represent the maximum agreed time that your work items should spend in a given process. The idea is to track whether your team is meeting their SLEs and continuously improve based on analyzing past cycle time data.

Setting Service Level Expectations can help a project function successfully when there's no binding contract.

Community management What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Manage OpenStack using Terraform and GitLab

Wed, 02/15/2023 - 16:00

Manage OpenStack using Terraform and GitLab ajscanlas Wed, 02/15/2023 - 03:00

One virtue of GitOps is Infrastructure as Code. It encourages collaboration by using a shared configuration and policy repository. Using GitLab can further enhance collaboration in your OpenStack cluster. GitLab CI can serve as your source control and orchestration hub for CI/CD, and it can even manage the state of Terraform.

To achieve this, you need the following:

GitLab account or instance.
Private OpenStack cluster. If you don't have one, read my article Set up OpenStack on a Raspberry Pi cluster.
A computer (preferably a container host).

GitLab and Terraform state

The goal is to achieve collaboration through Terraform, so you need to have a centralized state file. GitLab has a managed state for Terraform. With this feature, you can enable individuals to manage OpenStack collaboratively.

Create a GitLab group and project

Log in to GitLab, click on the hamburger menu, and click Groups→View all groups.

Image by:

(AJ Canlas, CC BY-SA 4.0)

Create a group by clicking on New group and then on Create group.

Image by:

(AJ Canlas, CC BY-SA 4.0)

Name the group to generate a unique group URL, and invite your team to work with you.

Image by:

(AJ Canlas, CC BY-SA 4.0)

After creating a group, create a project by clicking Create new project, and then Create blank project:

Image by:

(AJ Canlas, CC BY-SA 4.0)

Name your project. GitLab generates a unique project URL for you. This project contains the repository for your Terraform scripts and Terraform state.

Create a personal access token

The repository needs a personal access token to manage this Terraform state. In your profile, select Edit Profile:

Image by:

(AJ Canlas, CC BY-SA 4.0)

Click Access Token in the side panel to access a menu for creating an access token. Save your token because you can't view it again.

Image by:

(AJ Canlas, CC BY-SA 4.0)

Clone the empty repository

On a computer with direct access to your OpenStack installation, clone the repository and then change to the resulting directory:

$ git clone git@gitlab.com:testgroup2170/testproject.git $ cd testprojectCreate the backend .tf and provider file

Create a backend file to configure GitLab as your state backend:

$ cat >> backend.tf << EOF terraform { backend "http" { } } EOF

This provider file pulls the provider for OpenStack:

$ cat >> provider.tf << EOF terraform { required_version = ">= 0.14.0" required_providers { openstack = { source = "terraform-provider-openstack/openstack" version = "1.49.0" } } } provider "openstack" { user_name = var.OS_USERNAME tenant_name = var.OS_TENANT password = var.OS_PASSWORD auth_url = var.OS_AUTH_URL region = var.OS_REGION } EOF

Because you've declared a variable in the provider, you must declare it in a variable file:

$ cat >> variables.tf << EOF variable "OS_USERNAME" { type = string description = "OpenStack Username" } variable "OS_TENANT" { type = string description = "OpenStack Tenant/Project Name" } variable "OS_PASSWORD" { type = string description = "OpenStack Password" } variable "OS_AUTH_URL" { type = string description = "OpenStack Identitiy/Keystone API for authentication" } variable "OS_REGION" { type = string description = "OpenStack Region" } EOF

Because you're initially working locally, you must set those variables to make it work:

$ cat >> terraform.tfvars << EOF OS_USERNAME = "admin" OS_TENANT = "admin" OS_PASSWORD = "YYYYYYYYYYYYYYYYYYYYYY" OS_AUTH_URL = "http://X.X.X.X:35357/v3" OS_REGION = "RegionOne" EOF

These details are available on your rc file on OpenStack.

Initialize the project in Terraform

Initializing the project is quite different because you need to tell Terraform to use GitLab as your state backend:

PROJECT_ID="" TF_USERNAME="" TF_PASSWORD="" TF_STATE_NAME="" TF_ADDRESS="https://gitlab.com/api/v4/projects/${PROJECT_ID}/terraform/state/${TF_STATE_NAME}" $ terraform init \ -backend-config=address=${TF_ADDRESS} \ -backend-config=lock_address=${TF_ADDRESS}/lock \ -backend-config=unlock_address=${TF_ADDRESS}/lock \ -backend-config=username=${TF_USERNAME} \ -backend-config=password=${TF_PASSWORD} \ -backend-config=lock_method=POST \ -backend-config=unlock_method=DELETE \ -backend-config=retry_wait_min=5

To view the gitlab-project-id, look in the project details just above the Project Information tab in the side panel. It's usually your project name.

Image by:

(AJ Canlas, CC BY-SA 4.0)

For me, it's 42580143.

Use your username for gitlab-username. Mine is ajohnsc.

The gitlab-personal-access-token is the token you created earlier in this exercise. In this example, I use wwwwwwwwwwwwwwwwwwwww. You can name your-unique-state-name anything. I used homelab.

Here is my initialization script:

PROJECT_ID="42580143" TF_USERNAME="ajohnsc" TF_PASSWORD="wwwwwwwwwwwwwwwwwwwww" TF_STATE_NAME="homelab" TF_ADDRESS="https://gitlab.com/api/v4/projects/${PROJECT_ID}/terraform/state/${TF_STATE_NAME}"

To use the file:

$ terraform init \ -backend-config=address=${TF_ADDRESS} \ -backend-config=lock_address=${TF_ADDRESS}/lock \ -backend-config=unlock_address=${TF_ADDRESS}/lock \ -backend-config=username=${TF_USERNAME} \ -backend-config=password=${TF_PASSWORD} \ -backend-config=lock_method=POST \ -backend-config=unlock_method=DELETE \ -backend-config=retry_wait_min=5

The output is similar to this:

Image by:

(AJ Canlas, CC BY-SA 4.0)

Test the Terraform script

This sets the size of the VMs for my OpenStack flavors:

$ cat >> flavors.tf << EOF resource "openstack_compute_flavor_v2" "small-flavor" { name = "small" ram = "4096" vcpus = "1" disk = "0" flavor_id = "1" is_public = "true" } resource "openstack_compute_flavor_v2" "medium-flavor" { name = "medium" ram = "8192" vcpus = "2" disk = "0" flavor_id = "2" is_public = "true" } resource "openstack_compute_flavor_v2" "large-flavor" { name = "large" ram = "16384" vcpus = "4" disk = "0" flavor_id = "3" is_public = "true" } resource "openstack_compute_flavor_v2" "xlarge-flavor" { name = "xlarge" ram = "32768" vcpus = "8" disk = "0" flavor_id = "4" is_public = "true" } EOF

The settings for my external network are as follows:

$ cat >> external-network.tf << EOF resource "openstack_networking_network_v2" "external-network" { name = "external-network" admin_state_up = "true" external = "true" segments { network_type = "flat" physical_network = "physnet1" } } resource "openstack_networking_subnet_v2" "external-subnet" { name = "external-subnet" network_id = openstack_networking_network_v2.external-network.id cidr = "10.0.0.0/8" gateway_ip = "10.0.0.1" dns_nameservers = ["10.0.0.254", "10.0.0.253"] allocation_pool { start = "10.0.0.2" end = "10.0.254.254" } } EOF

Router settings look like this:

$ cat >> routers.tf << EOF resource "openstack_networking_router_v2" "external-router" { name = "external-router" admin_state_up = true external_network_id = openstack_networking_network_v2.external-network.id } EOF

Enter the following for images:

$ cat >> images.tf << EOF resource "openstack_images_image_v2" "cirros" { name = "cirros" image_source_url = "https://download.cirros-cloud.net/0.6.1/cirros-0.6.1-x86_64-disk.img" container_format = "bare" disk_format = "qcow2" } EOF

Here is a Demo tenant:

$ cat >> demo-project-user.tf << EOF resource "openstack_identity_project_v3" "demo-project" { name = "Demo" } resource "openstack_identity_user_v3" "demo-user" { name = "demo-user" default_project_id = openstack_identity_project_v3.demo-project.id password = "demo" } EOF

When complete, you will have this file structure:

. ├── backend.tf ├── demo-project-user.tf ├── external-network.tf ├── flavors.tf ├── images.tf ├── provider.tf ├── routers.tf ├── terraform.tfvars └── variables.tfIssue plan

After the files are complete, you can create the plan files with the terraform plan command:

$ terraform plan Acquiring state lock. This may take a few moments... Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # openstack_compute_flavor_v2.large-flavor will be created + resource "openstack_compute_flavor_v2" "large-flavor" { + disk = 0 + extra_specs = (known after apply) + flavor_id = "3" + id = (known after apply) + is_public = true + name = "large" + ram = 16384 + region = (known after apply) + rx_tx_factor = 1 + vcpus = 4 } [...] Plan: 10 to add, Releasing state lock. This may take a few moments...

After all plan files have been created, apply them with the terraform apply command:

$ terraform apply -auto-approve Acquiring state lock. This may take a few moments... [...] Plan: 10 to add, 0 to change, 0 to destroy. openstack_compute_flavor_v2.large-flavor: Creating... openstack_compute_flavor_v2.small-flavor: Creating... openstack_identity_project_v3.demo-project: Creating... openstack_networking_network_v2.external-network: Creating... openstack_compute_flavor_v2.xlarge-flavor: Creating... openstack_compute_flavor_v2.medium-flavor: Creating... openstack_images_image_v2.cirros: Creating... [...] Releasing state lock. This may take a few moments... Apply complete! Resources: 10 added, 0 changed, 0 destroyed.

After applying the infrastructure, return to GitLab and navigate to your project. Look in Infrastructure → Terraform to confirm that the state homelab has been created.

Image by:

(AJ Canlas, CC BY-SA 4.0)

Destroy the state to test CI

Now that you've created a state, try destroying the infrastructure so you can apply the CI pipeline later. Of course, this is purely for moving from Terraform CLI to a Pipeline. If you have an existing infrastructure, you can skip this step.

$ terraform destroy -auto-approve Acquiring state lock. This may take a few moments... openstack_identity_project_v3.demo-project: Refreshing state... [id=5f86d4229003404998dfddc5b9f4aeb0] openstack_networking_network_v2.external-network: Refreshing state... [id=012c10f3-8a51-4892-a688-aa9b7b43f03d] [...] Plan: 0 to add, 0 to change, 10 to destroy. openstack_compute_flavor_v2.small-flavor: Destroying... [id=1] openstack_compute_flavor_v2.xlarge-flavor: Destroying... [id=4] openstack_networking_router_v2.external-router: Destroying... [id=73ece9e7-87d7-431d-ad6f-09736a02844d] openstack_compute_flavor_v2.large-flavor: Destroying... [id=3] openstack_identity_user_v3.demo-user: Destroying... [id=96b48752e999424e95bc690f577402ce] [...] Destroy complete! Resources: 10 destroyed.

You now have a state everyone can use. You can provision using a centralized state. With the proper pipeline, you can automate common tasks.

Set up a GitLab runner

Your OpenStack cluster isn't public-facing, and the OpenStack API isn't exposed. You must have a GitLab runner to run GitLab pipelines. GitLab runners are services or agents that run and perform tasks on the remote GitLab server.

On a computer on a different network, create a container for a GitLab runner:

$ docker volume create gitlab-runner-config $ docker run -d --name gitlab-runner --restart always \ -v /var/run/docker.sock:/var/run/docker.sock \ -v gitlab-runner-config:/etc/gitlab-runner \ gitlab/gitlab-runner:latest $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 880e2ed289d3 gitlab/gitlab-runner:latest "/usr/bin/dumb-init …" 3 seconds ago Up 2 seconds gitlab-runner-test

Now register it with your project in your GitLab project's Settings → CI/CD panel:

Image by:

(AJ Canlas, CC BY-SA 4.0)

Scroll down to Runners → Collapse:

Image by:

(AJ Canlas, CC BY-SA 4.0)

The GitLab runner registration token and URL are required. Disable the shared runner on the right side to ensure it works on the runner only. Run the gitlab-runner container to register the runner:

$ docker exec -ti gitlab-runner /usr/bin/gitlab-runner register Runtime platform arch=amd64 os=linux pid=18 revision=6d480948 version=15.7.1 Running in system-mode. Enter the GitLab instance URL (for example, https://gitlab.com/): https://gitlab.com/ Enter the registration token: GR1348941S1bVeb1os44ycqsdupRK Enter a description for the runner: [880e2ed289d3]: dockerhost Enter tags for the runner (comma-separated): homelab Enter optional maintenance note for the runner: WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see https://gitlab.com/gitlab-org/gitlab/-/issues/380872 Registering runner... succeeded runner=GR1348941S1bVeb1o Enter an executor: docker-ssh, shell, virtualbox, instance, kubernetes, custom, docker, parallels, ssh, docker+machine, docker-ssh+machine: docker Enter the default Docker image (for example, ruby:2.7): ajscanlas/homelab-runner:3.17 Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! Configuration (with the authentication token) was saved in "/etc/gitlab-runner/config.toml"

Upon success, your GitLab interface displays your runner as valid. It looks like this:

Image by:

(AJ Canlas, CC BY-SA 4.0)

Explore the open source cloud Free online course: Developing cloud-native applications with microservices eBook: Modernize your IT with managed cloud services Try for 60 days: Red Hat OpenShift Dedicated Free online course: Containers, Kubernetes and Red Hat OpenShift What is Kubernetes? Understanding edge computing Latest articles for IT architects

You can now use that runner to automate provisioning with a CI/CD pipeline in GitLab.

Set up the GitLab pipeline

Now you can set up a pipeline. Add a file named .gitlab-ci.yaml in your repository to define your CI/CD steps. Ignore the files you don't need, like .terraform directories and sensitive data like variable files.

Here's my .gitignore file:

$ cat .gitignore *.tfvars .terraform*

Here are my CI pipeline entries in .gitlab-ci.yaml:

$ cat .gitlab-ci.yaml default: tags: - homelab variables: TF_ROOT: ${CI_PROJECT_DIR} TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/homelab cache: key: homelab paths: - ${TF_ROOT}/.terraform* stages: - prepare - validate - build - deploy before_script: - cd ${TF_ROOT} tf-init: stage: prepare script: - terraform --version - terraform init -backend-config=address=${BE_REMOTE_STATE_ADDRESS} -backend-config=lock_address=${BE_REMOTE_STATE_ADDRESS}/lock -backend-config=unlock_address=${BE_REMOTE_STATE_ADDRESS}/lock -backend-config=username=${BE_USERNAME} -backend-config=password=${BE_ACCESS_TOKEN} -backend-config=lock_method=POST -backend-config=unlock_method=DELETE -backend-config=retry_wait_min=5 tf-validate: stage: validate dependencies: - tf-init variables: TF_VAR_OS_AUTH_URL: ${OS_AUTH_URL} TF_VAR_OS_PASSWORD: ${OS_PASSWORD} TF_VAR_OS_REGION: ${OS_REGION} TF_VAR_OS_TENANT: ${OS_TENANT} TF_VAR_OS_USERNAME: ${OS_USERNAME} script: - terraform validate tf-build: stage: build dependencies: - tf-validate variables: TF_VAR_OS_AUTH_URL: ${OS_AUTH_URL} TF_VAR_OS_PASSWORD: ${OS_PASSWORD} TF_VAR_OS_REGION: ${OS_REGION} TF_VAR_OS_TENANT: ${OS_TENANT} TF_VAR_OS_USERNAME: ${OS_USERNAME} script: - terraform plan -out "planfile" artifacts: paths: - ${TF_ROOT}/planfile tf-deploy: stage: deploy dependencies: - tf-build variables: TF_VAR_OS_AUTH_URL: ${OS_AUTH_URL} TF_VAR_OS_PASSWORD: ${OS_PASSWORD} TF_VAR_OS_REGION: ${OS_REGION} TF_VAR_OS_TENANT: ${OS_TENANT} TF_VAR_OS_USERNAME: ${OS_USERNAME} script: - terraform apply -auto-approve "planfile"

The process starts by declaring that every step and stage is under the homelab tag, allowing your GitLab runner to run it.

default: tags: - homelab

Next, the variables are set on the pipeline. The variables are only present when the pipeline is running:

variables: TF_ROOT: ${CI_PROJECT_DIR} TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/homelab

There's a cache that saves specific files and directories upon running from stage to stage:

cache: key: homelab paths: - ${TF_ROOT}/.terraform*

These are the stages that the pipeline follows:

stages: - prepare - validate - build - deploy

This declares what to do before any stages are run:

before_script: - cd ${TF_ROOT}

In the prepare stage, the tf-init initializes the Terraform scripts, gets the provider, and sets its backend to GitLab. Variables that aren't declared yet are added as environment variables later.

tf-init: stage: prepare script: - terraform --version - terraform init -backend-config=address=${BE_REMOTE_STATE_ADDRESS} -backend-config=lock_address=${BE_REMOTE_STATE_ADDRESS}/lock -backend-config=unlock_address=${BE_REMOTE_STATE_ADDRESS}/lock -backend-config=username=${BE_USERNAME} -backend-config=password=${BE_ACCESS_TOKEN} -backend-config=lock_method=POST -backend-config=unlock_method=DELETE -backend-config=retry_wait_min=5

In this part, the CI job tf-validate and the stage validate run Terraform to validate that the Terraform scripts are free of syntax errors. Variables not yet declared are added as environment variables later.

tf-validate: stage: validate dependencies: - tf-init variables: TF_VAR_OS_AUTH_URL: ${OS_AUTH_URL} TF_VAR_OS_PASSWORD: ${OS_PASSWORD} TF_VAR_OS_REGION: ${OS_REGION} TF_VAR_OS_TENANT: ${OS_TENANT} TF_VAR_OS_USERNAME: ${OS_USERNAME} script: - terraform validate

Next, the CI job tf-build with the stage build creates the plan file using terraform plan and temporarily saves it using the artifacts tag.

tf-build: stage: build dependencies: - tf-validate variables: TF_VAR_OS_AUTH_URL: ${OS_AUTH_URL} TF_VAR_OS_PASSWORD: ${OS_PASSWORD} TF_VAR_OS_REGION: ${OS_REGION} TF_VAR_OS_TENANT: ${OS_TENANT} TF_VAR_OS_USERNAME: ${OS_USERNAME} script: - terraform plan -out "planfile" artifacts: paths: - ${TF_ROOT}/planfile

In the next section, the CI job tf-deploy with the stage deploy applies the plan file.

tf-deploy: stage: deploy dependencies: - tf-build variables: TF_VAR_OS_AUTH_URL: ${OS_AUTH_URL} TF_VAR_OS_PASSWORD: ${OS_PASSWORD} TF_VAR_OS_REGION: ${OS_REGION} TF_VAR_OS_TENANT: ${OS_TENANT} TF_VAR_OS_USERNAME: ${OS_USERNAME} script: - terraform apply -auto-approve "planfile"

There are variables, so you must declare them in Settings → CI/CD → Variables → Expand.

Image by:

(AJ Canlas, CC BY-SA 4.0)

Add all the variables required:

BE_ACCESS_TOKEN => GitLab Access Token BE_REMOTE_STATE_ADDRESS => This was the rendered TF_ADDRESS variable BE_USERNAME => GitLab username OS_USERNAME => OpenStack Username OS_TENANT => OpenStack tenant OS_PASSWORD => OpenStack User Password OS_AUTH_URL => Auth URL OS_REGION => OpenStack Region

So for this example, I used the following:

BE_ACCESS_TOKEN = "wwwwwwwwwwwwwwwwwwwww" BE_REMOTE_STATE_ADDRESS = https://gitlab.com/api/v4/projects/42580143/terraform/state/homelab BE_USERNAME = "ajohnsc" OS_USERNAME = "admin" OS_TENANT = "admin" OS_PASSWORD = "YYYYYYYYYYYYYYYYYYYYYY" OS_AUTH_URL = "http://X.X.X.X:35357/v3" OS_REGION = "RegionOne"

And it is masked GitLab for its protection.

Image by:

(AJ Canlas, CC BY-SA 4.0)

The last step is to push the new files to the repository:

$ git add . $ git commit -m "First commit" [main (root-commit) e78f701] First commit 10 files changed, 194 insertions(+) create mode 100644 .gitignore create mode 100644 .gitlab-ci.yml create mode 100644 backend.tf create mode 100644 demo-project-user.tf create mode 100644 external-network.tf create mode 100644 flavors.tf create mode 100644 images.tf create mode 100644 provider.tf create mode 100644 routers.tf create mode 100644 variables.tf $ git push Enumerating objects: 12, done. Counting objects: 100% (12/12), done. Delta compression using up to 4 threads Compressing objects: 100% (10/10), done. Writing objects: 100% (12/12), 2.34 KiB | 479.00 KiB/s, done. Total 12 (delta 0), reused 0 (delta 0), pack-reused 0 To gitlab.com:testgroup2170/testproject.git * [new branch] main -> mainView the results

View your new pipelines in the CI/CD section of GitLab.

Image by:

(AJ Canlas, CC BY-SA 4.0)

On the OpenStack side, you can see the resources created by Terraform.

The networks:

Image by:

(AJ Canlas, CC BY-SA 4.0)

The flavors:

Image by:

(AJ Canlas, CC BY-SA 4.0)

The images:

Image by:

(AJ Canlas, CC BY-SA 4.0)

The project:

Image by:

(AJ Canlas, CC BY-SA 4.0)

The user:

Image by:

(AJ Canlas, CC BY-SA 4.0)

Next steps

Terraform has so much potential. Terraform and Ansible are great together. In my next article, I'll demonstrate how Ansible can work with OpenStack

Follow this tutorial to see how using GitLab can further enhance collaboration in your OpenStack cluster.

Image by:

Opensource.com

Containers OpenStack Git GitLab Cloud CI/CD Automation What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

How I use Ansible to add a feature to my Linux KDE desktop

Wed, 02/15/2023 - 16:00

How I use Ansible to add a feature to my Linux KDE desktop sethkenlon Wed, 02/15/2023 - 03:00

I run the KDE Plasma Desktop on my computer because it's a flexible environment with lots of options for customization. Having choices in your desktop is about more than just having lots of menus and buttons to activate or deactivate. The thing I love most about KDE Plasma Desktop is the ability to add my own features to it. One reason this is possible is KServices, a simple but powerful plugin framework for handling desktop services.

Add functions to the right-click menu

In the KDE Plasma Desktop, there's usually a contextual menu available when you right-click on something, such as a directory or a file. You can add items to the right-click menu by creating your own KService, and you don't need anything more than a rudimentary understanding of Bash to make it work.

First, create a new directory for your service menu:

$ mkdir -p ~/.local/share/kio/servicemenus

Add a .desktop file:

$ touch ~/.local/share/kio/servicemenus/hello.desktop

Open the hello.desktop file in a text editor. A .desktop file is a small configuration file used by the menu system of the Linux desktop. Here's a simple KServices file that generates a hello.txt file in the directory you select:

[Desktop Entry] Type=Service MimeType=inode/directory; Actions=Hello [Desktop Action Hello] Name=Say hello Icon=/usr/share/icons/breeze-dark/actions/symbolic/file-library-symbolic.svg Exec=touch %f/hello.txt

The first configuration block tells your system that this .desktop file is a service rather than, for instance, an application.
The MimeType tells the Plasma Desktop to only show this action as an option when you right-click on a folder, not a file.
The Actions line identifies what action is taken when this service is activated. The name Hello is arbitrary, and refers to the next configuration block, which is a Desktop Action configuration with the name Hello.

Here's what the next configuration block means:

The Desktop Action definition named Hello.
The values for Name and Icon appear in the right-click menu.
The Exec line is the command you want to run when your service is selected. As a simple demonstration, this .desktop file just creates an empty file called hello.txt in the location that you right-clicked on (represented by the special variable %f).

Save the .desktop file, and then make it executable:

$ chmod +x ~/.local/share/kio/servicemenus/hello.desktop

Start a new instance of the Dolphin file manager and create an empty folder. Then right-click on the folder and navigate to Actions. In the Actions menu, there's a new service available, and it's called Say hello because that's what your .desktop file has set in the Name field.

Image by:

(Seth Kenlon, CC BY-SA 4.0)

Look in the folder after running the service to see the empty hello.txt file that's been created.

Mimetypes

This sample KService only works on directories because the .desktop file defining the service specifies the Mimetype: inode/directory. That's what tells Dolphin not to display the service when you right-click on a file.

Using mimetypes, you can create highly specific services based on what kind of file system object you select when you right-click. The perl-file-mimeinfo package provides a mimetype command, which you can use to get the mimetype of any file. Install it with your distribution's package manager, and then try it out:

$ mimetype example.txt example.txt: text/plain $ mimetype Photos/example.webp Photos/example.webp: image/webp

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Executables

I demonstrated a simple "hello world" example in this article, but KServices can be as complex as you need them to be. The Exec line of your KService .desktop file can launch any application or script, and the %f variable ensures that the target or destination of whatever gets launched is what you've right-clicked on.

For my own workflow, I used to use Planter to quickly construct a project environment. Lately, though, I've switched to Ansible and this KService:

[Desktop Entry] Type=Service MimeType=inode/directory; Actions=planter [Desktop Action planter] Name=Create project directory Icon=/usr/share/icons/breeze-dark/actions/symbolic/folder-new-symbolic.svg Exec=ansible-playbook /home/seth/Ansible/playbooks/standard_dirs.yaml -e dir=%f

Here's my Ansible playbook:

--- - hosts: localhost tasks: - name: Create directories ansible.builtin.file: path: "{{ item }}" state: directory with_items: - '{{ dir }}/video' - '{{ dir }}/edit' - '{{ dir }}/audio' - '{{ dir }}/title' - '{{ dir }}/render' - '{{ dir }}/graphic' - '{{ dir }}/photo'

When I right-click on a directory and select Create project directory, the subdirectories I need for media projects are added to that directory. It's a simple feature for a desktop, and a little unique to a specific workflow, but it's the feature I want. And thanks to KServices, it's a feature I have. Try out KServices in the KDE Plasma Desktop for yourself, and add the feature you want.

Follow this tutorial to see how I use KService and Ansible on my Linux KDE desktop.

Linux Ansible What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Create a modern user interface with the Tkinter Python library

Tue, 02/14/2023 - 16:00

Create a modern user interface with the Tkinter Python library patrik-dufresne Tue, 02/14/2023 - 03:00

Python's Tkinter library isn't exactly known for its good looks. I've developed a library to help create a modern graphical user interface for Python.

I spent a lot of time searching for a simple but modern GUI toolkit before developing a new library called TKVue that creates graphical interfaces for desktop applications. Through my research, I realized that there were several different libraries to create graphical interfaces. However, most involve adding new dependencies to bind with graphical libraries. For example, there's a library for Qt, another for wxWidgets, and a third for GTK. None are native to Python or entirely coded in Python. That's a problem. If you want to code a GUI with Qt, it's necessary to compile the Qt source code on each platform you want to support. I wanted to target the three leading platforms: Linux, Windows, and Mac.

The big advantage of Tkinter is that it's embedded in Python. There's no need for new dependencies or to compile new libraries. Everything's already done for you.

In short, it is best to use Tkinter to create something portable.

Tkinter with a modern theme

Creating a GUI with Tkinter is easy, but there's no denying that by default, it looks like a GUI from the 1980s. In addition to creating graphical interfaces that aren't very pleasing to the eye, the programming methodology is also from the 1980s: Programming the Tkinter graphical interface is not declarative.

Motivated by Tkinter's portability, I was determined to use it to create a professional and modern graphical interface. My research led me to discover a whole system for modifying Tkinter's appearance using themes. Tkinter themes are similar to the CSS file of a web page. They allow you to configure the appearance of the components that make up the graphical interface by declaring new styles. The creation of these styles requires some work, but the system is flexible enough to allow the creation of a modern-looking graphical interface. This work is similar to customizing a CSS theme in web development. If you create a web page without CSS, the appearance is not modern, and a lot of work is needed to improve it. This is why CSS libraries such as bootstrap are used to speed up the creation of the graphic interface.

In the Tkinter universe, there is no CSS library. Some pre-existing themes exist, but it's preferable for any project to customize the color palette to match your product branding and give it a web look-and-feel.

To achieve that, the most important element in creating a modern interface with Tkinter is changing the background color and the buttons' appearance.

Once properly personalized to your liking, the result is a clean and visually attractive graphical interface.

Here is the "default" theme:

Image by:

(Patrik Dufresne, CC BY-SA 4.0)

The "clam" theme looks like this:

Image by:

(Patrik Dufresne, CC BY-SA 4.0)

Then with my personalization:

Image by:

(Patrik Dufresne, CC BY-SA 4.0)

TKVue:

import tkvue import tkinter.ttk as ttk tkvue.configure_tk(theme="clam") class RootDialog(tkvue.Component): template = """ """ def __init__(self, master=None): super().__init__(master) s = ttk.Style(master=self.root) s.configure('H1.TLabel', font=['Lato', '-60'], background='#ffffff') s.configure('default.TFrame', background='#ffffff') s.configure( default.TButton', foreground='#0E2933', background='#B6DDE2', bordercolor='#ACD1D6', darkcolor='#B6DDE2', lightcolor='#B6DDE2', focuscolor='#0E2933', ) s.map( default.TButton', background=[('disabled', '#E9F4F6'), ('hover !disabled', '#9ABBC0'), ('pressed !disabled', '#88A5A9')], ) if __name__ == "__main__": dlg = RootDialog() dlg.mainloop()Use a declarative language

After discovering the Tkinter style system, another problem to solve is using a declarative language. For a person who develops web interfaces every day, creating a GUI using Python code doesn't make sense. It requires too much code and too many variables, and poorly structured code. Creating a simple button requires many lines of code intertwined through the creation of many other components.

Modifying the GUI is tedious because it requires many variables that are all tangled up.

In comparison, creating a graphical interface using HTML tags is much more declarative and allows the creation of complex graphical interfaces while remaining structured. That's why I decided to use the Markup Language to create a graphical interface easily from the components available in Tkinter.

The initial idea was quite simple; a tag corresponds to a component. If I want to create a button component with the text Hello, I just have to make a tag named button with an attribute with the value Hello.

TKVue:

Python:

b = Button(parent, text ="Hello") b.pack()

I extrapolated the concept to all the components and functionalities in the Tkinter library from this starting idea. So, to create a more complex graphical interface with different elements, just nest them inside each other by creating a parent-child structure and defining the attributes for each component.

TKVue:

Python:

import tkinter from tkinter.ttk import Frame, Label, Combobox top = tkinter.Tk() top.title = "TKVue Test frame = tkinter.Frame(top) frame.pack(fill='both', expand=1, padx=10, pady=10) l = Label(frame, text="Available values: ", width=20) l.pack(side='left') c = Combobox(frame, values=['zero', 'one', 'two', 'three']) c.pack(side='left', expand=1) top.mainloop()

Result:

Image by:

(Patrik Dufresne, CC BY-SA 4.0)

Using a hierarchical design similar to HTML simplifies the creation of the graphical interface and significantly simplifies the presentation of the parent-child structure created between each component.

Create a dynamic interface

The other problem I encountered using Tkinter was the non-dynamic side of the interface. Having coded Web GUIs with Jinja2 templates, which allow you to reuse variables and create loops easily, I wanted similar functionality when creating desktop GUIs.

I must admit that the dynamic side of TKVue gave me some problems. My initial plan was to update the GUI dynamically when a variable was changed by the code. So it should be possible to use variables in the markup language. Suppose I select a value from a drop-down list. I want to be able to associate the specified value with a variable and allow that variable to be reused in another component for display. See the code sample below:

TKVue:

import tkvue class RootDialog(tkvue.Component): template = """ """ data = tkvue.Context({"myvalues": ["zero", "one", "two", "three"], "var1": ""}) dlg = RootDialog() dlg.mainloop()

Python:

import tkinter from tkinter.ttk import Frame, Label, Combobox top = tkinter.Tk() top.title = "TKVue Test frame = tkinter.Frame(top) frame.pack(fill='both', expand=1, padx=10, pady=10) l = Label(frame, text="Available values: ", width=20) l.pack(side='left') var1 = tkinter.StringVar() c = Combobox(frame, values=['zero', 'one', 'two', 'three'], textvariable=var1) c.pack(side='left', expand=1) frame2 = tkinter.Frame(top) frame2.pack(fill='both', expand=1, padx=10, pady=10) s = Label(frame2, text='Value selected:') s.pack(side='bottom') var1.trace_add("write", lambda *args: s.configure(text='Value selected: ' + var1.get())) top.mainloop()

Result:

Image by:

(Patrik Dufresne, CC BY-SA 4.0)

With this in place, it is possible to create a graphical interface that reacts to the user's actions in a simple and declarative way.

Use loops and conditions

To make the model as dynamic as possible, it was necessary to support the use of loops and conditions to show or hide components depending on variables. TKVue introduces two special attributes to fill this need, the for attribute for loop creation and the visible attribute.

The following example allows the user to select the number of items to be displayed in the loop from a drop-down list. The following example demonstrates the dynamic side of a loop reacting to the user's actions:

TKVue:

import tkvue class RootDialog(tkvue.Component): template = """ """ data = tkvue.Context({'count': 5}) if __name__ == "__main__": dlg = RootDialog() dlg.mainloop()

Result:

Image by:

(Patrik Dufresne, CC BY-SA 4.0)

The condition declaration is even simpler. You must define an expression in the visible attribute as follows:

TkVue:

import tkvue class RootDialog(tkvue.Component): template = """ """ data = tkvue.Context({'show': True}) if __name__ == "__main__": dlg = RootDialog() dlg.mainloop()

Result:

Image by:

(Patrik Dufresne, CC BY-SA 4.0)

More Python resources What is an IDE? Cheat sheet: Python 3.7 for beginners Top Python GUI frameworks Download: 7 essential PyPI libraries Red Hat Developers Latest Python articles Use localization

Another important aspect of the development of my application is to allow the translation of the graphical interface quickly by using technologies that are already known and widely used in Python: gettext.

To do this, the library provides an adapter for Babel, which allows extracting the strings to be translated. You can then find these strings in a translation catalog (.po file).

Here is an example of a configuration file for Babel that allows you to extract strings for translation purposes:

Place the following in Babel.cfg:

[python: **.py] [tkvue: **/templates/**.html]

I'm using Tkinter and my theming library for my data backup software, Minarca.

The final result is elegant and simple at the same time. The style matches the color palette of the Minarca project.

Image by:

(Patrik Dufresne, CC BY-SA 4.0)

Try TKVue today

The objective of all this work is to bring familiar advantages in web development to traditional development. TKVue saves time while developing the graphical interface and makes it easier to maintain and modify the code afterward.

The work I have done so far in the library is quite rudimentary and needs further modification before being used more widely in projects. I hope that the publication of this article allows someone else to take over the development of a similar library or to continue to develop directly in TKVue to improve its functionality. TKVue's code is available on GitLab.

TKVue saves time while developing the graphical interface and makes it easier to maintain and modify the code afterward.

Image by:

Opensource.com

Python Web development What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Lua loops: how to use while and repeat until

Tue, 02/14/2023 - 16:00

Lua loops: how to use while and repeat until sethkenlon Tue, 02/14/2023 - 03:00

Control structures are an important feature of programming languages because they enable you to direct the flow of the program based on conditions that are often established dynamically as the program is running. Different languages provide different controls, and in Lua there's the while loop, for loop, and repeat until loop. This article covers the while and repeat until loops. Because of their flexibility, I cover for loops in a separate article.

A condition is defined by an expression using an operator, which is a fancy term for symbols you may recognize from math classes. Valid operators in Lua are:

== equal to
~= not equal to
< less than
> greater than
⇐ less than or equal to
>= greater than or equal to

Those are known as relational operators because they prompt an investigation of how two values relate to one another. There are also logical operators, which mean the same as they mean in English and can be incorporated into conditions to further describe the state you want to check for:

and
or

Here are some example conditions:

foo > 3: Is the variable foo is greater than 3? The foo must be 4 or more to satisfy this condition.
foo >= 3: Is foo greater than or equal to 3? The foo must be 3 or more to satisfy this condition.
foo > 3 and bar < 1: Is foo greater than 3 while bar is less than 1? For this condition to be true, the foo variable must be 4 or more at the same moment that bar is 0.
foo > 3 or bar < 1: Is foo greater than 3? Alternately, is bar less than 1? If foo is 4 or more, or bar is 0, then this condition is true. What happens if foo is 4 or more while bar is 0? The answer appears later in this article.

While loop

A while loop executes instructions for as long as some condition is satisfied. For example, suppose you're developing an application to monitor an ongoing zombie apocalypse. When there are no zombies remaining, then there is no more zombie apocalypse:

zombie = 1024 while (zombie > 0) do print(zombie) zombie = zombie-1 end if zombie == 0 then print("No more zombie apocalypse!") end

Run the code to watch the zombies vanish:

$ lua ./while.lua 1024 1023 [...] 3 2 1 No more zombie apocalypse!Until loop

Lua also has a repeat until loop construct that's essentially a while loop with a "catch" statement. Suppose you've taken up gardening and you want to track what's left to harvest:

mytable = { "tomato", "lettuce", "brains" } bc = 3 repeat print(mytable[bc]) bc = bc - 1 until( bc == 0 )

Run the code:

$ lua ./until.lua brains lettuce tomato

That's helpful!

Infinite loops

An infinite loop has a condition that can never be satisfied, so it runs infinitely. This is often a bug caused by bad logic or an unexpected state in your program. For instance, at the start of this article, I posed a logic puzzle. If a loop is set to run until foo > 3 or bar < 1, then what happens when foo is 4 or more while bar is 0?

Here's the code to solve this puzzle, with a safety catch using the break statement just in case:

foo = 9 bar = 0 while ( foo > 3 or bar < 1 ) do print(foo) foo = foo-1 -- safety catch if foo < -800000 then break end end

Programming and development Red Hat Developers Blog Programming cheat sheets Try for free: Red Hat Learning Subscription eBook: An introduction to programming with Bash Bash shell scripting cheat sheet eBook: Modernizing Enterprise Java An open source developer's guide to building applications

You can safely run this code, but it does mimic an accidental infinite loop. The flawed logic is the or operator, which permits this loop to continue both when foo is greater than 3 and when bar is less than 1. The and operator has a different effect, but I leave that to you to explore.

Infinite loops actually do have their uses. Graphical applications use what are technically infinite loops to keep the application window open. There's no way of knowing how long your user intends to use the application, so the program runs infinitely until the user selects Quit. The simple condition used in these cases is one that is obviously always satisfied. Here's an example infinite loop, again with a safety catch built in for convenience:

n = 0 while true do print(n) n = n+1 if n > 100 then break end end

The condition while true is always satisfied because true is always true. It's the terse way of writing while 1 == 1 or something similarly eternally true.

Loops in Lua

As you can tell from the sample code, although there are different implementations, loops all basically work toward the same goal. Choose the one that makes sense to you and that works best with the processing you need to perform. And just in case you need it: The keyboard shortcut to terminate a runaway loop is Ctrl+C.

Learn how and when to use while and repeat until loops in Lua.

Image by:

WOCinTech Chat. Modified by Opensource.com. CC BY-SA 4.0

Programming What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

A 10-step guide for a successful hackathon

Mon, 02/13/2023 - 16:00

A 10-step guide for a successful hackathon Tiffany Long Mon, 02/13/2023 - 03:00

Hackathons are easy. How much thought do you need to put into them anyway? Just set a date, and people will show up. Well, that is not quite true!

While you may get lucky with that approach, the reality is that hackathons are a keystone experience in the tech industry, and attendees have specific expectations. Not only that, but your organization also has certain needs and should set goals for a hackathon. So, how do you ensure that a hackathon works for your organization and attendees?

A successful hackathon depends on several decisions that tend to be recursive. Decisions about what you want to achieve will impact what resources you allot and how you want to communicate. Those decisions affect whether you go virtual or in person, and that decision will once again impact the resources you need and how you communicate. Alignment when planning hackathons is not just about getting people to agree. You will have a whole suite of decisions that must internally align. For example, a technically difficult hackathon might not be able to attract a large audience (ask me how I know!) and will require a specialized recruitment strategy that requires different resources.

I've done many hackathons over the years, including just a few months back, when my organization hosted a hackathon that led to new features that we will incorporate into the next version of our open source product, Traefik Proxy 3.0. So, trust me when I say planning a hackathon that will enrich attendees and create valuable outcomes for your project is about more than hope, pizza, and chaos.

This article uses the most recent Traefik Labs Hackathon as a blueprint. I share a checklist, tips, and tricks to help you identify your objectives, plan, manage the contest and compensation, share your results, and manage the long tail of the hackathon (the work isn't over when the hackathon ends!)

This guide serves as a model for you to outline best practices so that you, too, can hold a successful hackathon with a sizable target audience that delivers results!

[ Get a PDF and EPUB version of this article. Download it here. ]

1. Three questions to determine your goals

The first and most crucial step is to set your goals. But this is no simple affair. Before you set goals, you need to coordinate internally on multiple fronts and ask questions such as:

Why do you want to do a hackathon?
Who do you want to attend?
How are you going to measure your success?

Identify your internal stakeholders and set expectations

Hackathons are cross-functional. No hackathon is run by a community person alone. It is important to ensure everyone is aligned on the goals, what is required to achieve them, and that the necessary resources are committed. This probably sounds super corporate, but these functions exist even within the smallest projects. A project needs adoption and code. It also needs value decisions based on who will be using it. And, of course, projects need passionate contributors.

Hackathons require cross-functional resources. One team with a single set of resources cannot successfully run a hackathon. The organization must make various resources available, including:

Marketing for planning and outreach.
Product Management for product and industry-specific insight.
Engineering for deep technical knowledge and community engagement.

For these reasons, hackathons usually support cross-functional goals. Your Community Team, for example, might want to build ownership and convert users to active community members. The Marketing Team might want to enhance awareness and court new users. The Engineering Team might need new perspectives on specific needs or challenges. The Product Team might have goals or no-go areas the community should be aware of.

And last but not least, the hackathon budget is cross-functional. I am sorry to inform you, but hackathons ain't free! Your largest expense is always the dedicated time of your team.

2. Why are you doing this?

Setting your goals is the most important part of a successful hackathon. If you don't know what you want to do or why a hackathon is important, at best, it will have a ton of wasted potential and be a disconnected mess at worst.

Communities feed off of ownership. Decide what you need from your community and what ownership stake you want community members to have. Without a clear understanding of this, your hackathon might not reach its full potential in empowering your community.

Be very careful with your hackathon design and goals. Different types of hackathons appeal to different skill levels. If the code you're looking for is very advanced, take the extra time to court the right audience and accept that it will include less overall attendance. Cast a wide net if the contributions can vary in skill and experience.

Are you hosting a hackathon to get code and build your project?

Sometimes, projects hit a critical juncture or acquire a lot of excitement around them, and you want to harness the energy to build something together. A hackathon is a great way to achieve this!
If you have an active community of users, a hackathon can bring everyone together at the same time to harness that excitement to feed the creative energy of your group.

Note: This is more easily achievable with smaller groups who know each other and have a shared experience with the project. You also need to carefully evaluate the skills required to build your project.

Are you hosting a hackathon to build your community or re-engage them?

Maybe you are just building your community or noticed that your community needs a little juice. Hackathons are exciting, and they can help bring that back.
Above, I said, "Communities feed off of ownership." If your community members do not feel they have a stake or that their needs and voices matter, they will drift away. This is common when projects grow and become more formalized. As the barrier to entry rises, the ability for community members to feel ownership falls, and the project becomes like a product to the user. One way to enhance community membership is by creating events that engage users and lower the bar for entry: Bug round-ups, light requests, and longer timelines.
Perhaps your user community is growing, but the contributor community is becoming more specialized as your tech becomes more complex. In this case, you need to court sophisticated technologists who understand your tech and the use cases. Look for community members who use your tech in their jobs—especially at companies with large or complex deployments. These people are more likely to understand the needs of users and of the tech itself. They will also have suggestions for significant and valuable enhancements.
You are free to choose goals that build your community and match your team and community members' energy and time. For example, at Traefik Labs, a hackathon aimed at enthusiastic folks with a small time commitment might target our Plugin Catalog. However, when looking for larger contributions or contributions that take significant expertise, we might target advanced technologists–especially those we know.

Are you hosting a hackathon to celebrate something?

Hackathons are a great way to celebrate a new launch and hype your community. For example, that is exactly why we hosted the Traefik Proxy 3.0 Hackaethon.
Hackathons are also great for getting the word out about a new product capability. The Traefik Plugin Hackaethon is an excellent example here.
Maybe you want to organize an event to celebrate your top contributors. Do it with a hackathon! Take a look at this hackathon organized by HackerOne. And if you're thinking, "but this is not about open source software (OSS), how can it be a hackathon?" I've got news for you—hackathons are not just for OSS! Hackathons are for creating with a large community.

Are you hosting a hackathon to build awareness?

Hackathons are a great place to begin if you are just starting and want to build awareness around your product/brand. However, there are a few conditions.

Laser-focused goals and big contributions are unlikely to happen at this stage. Go for a softer, broader focus, and minimize the work required by attendees.
Reach out to new community members, less experienced users, and users with less exposure to your specific project.

Are you hosting a hackathon to connect to users?

I can think of no better way to connect new users to your project than a hackathon. Not only will your users become intimately familiar with your project, but hackathons also have a unique way of engendering a sense of ownership, rarely achievable through other types of events.

3. Who is your audience?

Assuming you have pinpointed why you want to host a hackathon and what you want to achieve, it's time to assess the characteristics that a participant needs to be successful. Use your decisions about your goals to identify your audience to ask what type of community member can help you achieve your objectives. Use the list of comparisons below:

Highly-skilled vs. mixed-skilled vs. low-skilled
Specialized vs. generalized skill
Intensive time vs. less intensive time
Individual contributions vs. group contributions

Your most active community members must look a bit like your target audience.

You might rethink your goals if your target audience doesn't align with at least 80% of the people you know you can attract. Accurately identifying your target audience will go a long way to making your communication strategy around the hackathon and the hackathon itself more successful.

4. How are you measuring goals?

Perfect, now that you answered the first two big questions and have your goals laid down, it's time for the third big question—how will you measure those goals? Inspiring your internal teams and your community to work together in building the future of your project, engendering ownership, and increasing engagement are awesome, but you can't determine success if you can't measure your goals.

What does success look like immediately after the event?

Clearly, the first sign of success is that your attendees had an overall good experience and are motivated to engage more with your project.
If you are looking for outreach, set a quantity of participants to shoot for and a number of participants who return to contribute more after the event or in three months.
- A major sign of success is whether attendees connect and engage with each other, co-educate, and build teams during their hackathon.
- Were mentorships built? Through partnership, did several newer users grow into skilled mid-level users, or did mid-level users evolve into expert-tier users? This is the gold ring of success indicators.
- Did your partner organizations (maybe universities) request future hackathons or other events?
If building awareness, you might also look for successful follow-up chatter. Who wrote blog posts? Were attendees talking about it on social media?
If you are looking for contributions, did they work for you? Are these the contributions you want? Did they impact how your team thinks about the problems they face? Will you have ongoing collaborations with these contributors?

What will denote success three months after the event?

Defining benchmarks for long-term success is just as important. Here are a few examples of what could indicate long-term success:

Your hackathon should increase the number of returning contributors to your project. The goal is to get people hooked. If people new to your project came from the hackathon and stayed as users, or if your existing users became more active, you know you won.
Hackathons are great as self-contained events, but they are supremely valuable as marketing content. They build trust in the community, showing you are responsive and value community input. They are fun loci of activity that let community members bond and look forward to the future, and they are aspirational. People love to see others celebrated and plan to achieve that celebration in the future.
When you build marketing content around your hackathon (or better yet, others build content around your hackathon), you can expand your reach among second-degree connections.
Tall poppy syndrome is a shame. Hackathons are a great opportunity to gather those participants who stood out and galvanize them to do other cool things and spread the word about your project.

5. Decide on in-person vs. virtual

I know what you're thinking—is in-person even a consideration? We've all gotten so used to doing everything virtually in the post-covid world. So, are the days of in-person gone? I would argue no, they are not. With care and safety measures in place, in-person events are the heart and soul of hackathons.

In-person means no distractions, lots of pizza, and energy drink-fueled friendship.
In-person fuels group participation rather than individual contributor participation.
In-person works well at scale and in miniature: Organizing in-person hackathons for large groups brings high energy and rewards. But they can get quite costly. If you want to organize a large-scale hackathon, you'll be more successful if you target less experienced developers (students, clubs, new careerists) because these folks have the most time and the most to gain when demonstrating their skill and passion.
In-person also works well for small groups and is great for intense planning and iteration—long nights with new and old friends, usually over food and beer!

And while many pros come with in-person hackathons, it doesn't mean that the virtual experience only comes with cons. Granted, nothing replaces that feeling of late nights with pizza, off-the-cuff remarks that end up changing your entire project, and a friendly set of eyes over your shoulder as you test or debug. But...

Virtual means you can get a wider group of participants at a significantly lower cost.
Virtual respects disability.
Virtual is geolocation friendly.
Virtual allows for higher individual contributor participation.
Virtual offers more flexibility in the style and length of the event – you cannot have a month-long in-person event!

Timelines of virtual hackathons

Did you decide to do a virtual hackathon? Great! Now, you need to determine the type of virtual hackathon you want. Do you envision a prolonged or intensive timeline? Keep in mind that the type of virtual hackathon you choose will determine, to some extent, your target audience and communication strategy.

Extended timeline:

Allows after-hours tinkering and enables developers to attend without taking time off from work.
Provides more time to solicit contributions.
Requires fewer resources for both the organizer and the participants.
Extended timelines require fewer real-time resources.

Intense timeline:

Recreates that feeling of intensity usually experienced in in-person hackathons.
Requires a high amount of resources for a short period of time.
Requires tight management and a communication platform.
Requires clear one-on-one communication, but also fosters group-to-group or intra-community communication.

6. Build your communication strategy

Speaking of communication, once you have your goals, you must decide who communicates with participants and how. It's common to choose between the popular apps of the day. Your choice impacts the event's feel. Different chat applications and collaboration platforms have their own cultures and strengths. The decision you made early on about how to host your hackathon (in-person or virtual, prolonged or intense timeline) is likely to have the most significant impact on your communication strategy.

In-person communication plan

If you are running an in-person hackathon, consider it a genuine event—it feels almost like a conference. In-person hackathons often include the following:

Workshops/round tables: Meant to educate and develop new industry standards/best practices for the concerns of the day. These sessions can function as proctored time-bound conversations amongst 6-10 individuals, where they agree upon findings and take notes that are made public to all participants.
Planning sessions: Often used for projects with non-code outcomes, like developing updated standards.
Coding sessions: Used for code-based projects which require work to maintain and enhance.

Each of the above has different communication needs:

General communication for running the event (food, cleaning, management of resources).
Masters of ceremonies to move through the agendas.
For workshops:
- People prepared to facilitate, but not lead, conversations in workshops.
- Note takers and people to make sure that the notes are turned into a publishable product.
- Project managers to ensure the above tasks are done.

Making this all happen requires the resources and specialized knowledge from your Community, Product Managers, and teach-savvy teams. From past experience, it took a team of community members and staff to manage an event of this scope. To be successful, your team will need specialized people as well.

You also need to decide what types of communication you want to foster and who is responsible for it:

Multiple teams will need to take shifts to perform general support.
A DevRel, engineering, or support team will need to manage technical communication between triage and participants.
Community Teams usually spend extensive time connecting participants to help build strong groups by reinforcing skills or points of view. This is one way to ensure that hackathon magic.
Community Teams also need to support marketing efforts to engage participants and manage follow-up.

Virtual communication plan

For virtual hackathons, the choice of a communication platform depends heavily on the outcome you want to achieve, the timeline you've chosen for your hackathon (prolonged or intensive), and the type of communication you wish to facilitate (synchronous or asynchronous).

Using Pull Requests and Issues on Git hosts (asynchronous):

Choosing to communicate through Git pull requests and Issues on your project directly frees up technical staff resources because it keeps the conversations about projects in your current process and allows your team to be responsive rather than instigating communication.
This approach is great if the team for your hackathon is small or if the expected contributions are relatively small and you do not plan to help participants form teams.
Using your existing processes is especially great for prolonged hackathons as they do not require additional moderation or require your team to monitor an additional app.
The downside is that you will only facilitate communication with the individual contributor or group of contributors already working together. It's difficult to connect participants who are working separately. Participants can't find each other as easily on their own, so you lose some of the magic that happens when hackathon participants organically talk to each other in open threads.

Using a chat application (synchronous):

Choosing dedicated chat servers is required for intense hackathons.
Chat facilitates the team formation and communication necessary for complex projects with fast timelines and sparks the brainstorming that preludes an awesome contribution.
Additionally, your goal is to build community. People want to join communities where they have ownership, have friends, and feel comfortable. You need a place for participants to feel connected to each other if you want them to return.
Chat servers can outlast an event, allowing for continued community engagement.

Regardless of which platform you choose, you need a communication plan that identifies when every person on your team is available. Managing a virtual hackathon can get quite tricky, primarily due to the different timezones—people can participate whenever they want, from wherever they want. You must plan to accommodate participants across all time zones and for every occasion. Draw up a plan with who is responsible (and when) for the following:

Determining response SLAs.
Animating your virtual space (a dead space guarantees poor communication).
Encouraging team building.
Responding to technical questions.
Checking in on participants.
Moderating the space to ensure the safety of your participants.

7. Decide on the prizes

Is your hackathon a contest? Hackathon participants are often content with grand prizes and "swagpaloozas" for top contributions. But before you decide on the fun stuff (the actual awards), you must determine what your contest values.

What differentiates a good contribution from a great contribution? If your attendees know how you feel about this, they are more likely to hit it out of the park.
What do you value? This is your chance to tell participants what you want to see submitted by attaching a prize to it. For example, during the last Traefik Hackaethon, we offered bounties for the most-wanted features. These were, indeed, the ones most people worked on.
Are there categories of contributions? You need to decide on prizes for each category.
Create a rubric (a chart or grid defining and ranking achievements, like this example). This way, participants know what you value and how they are judged. This was one way we improved submissions at HackerOne.

On the other hand, some may argue that competition is overrated. If your goal is participation, feel free to reward every single one of your participants for simply giving back! Hacktoberfest is a great example of this approach.

8. Swag it up

Everyone loves swag! And your participants would certainly appreciate a token to remember this event, whether virtual or in person. Swag has two purposes:

Swag shows your appreciation: The contributors took their time to engage with you in an intense way; thank them with a gift that shows you value their contributions.
Swag builds awareness: Gifting swag to your participants helps them spread the love and build awareness of your community by sharing their loot and experience.

The community loves swag, but they don't love boring swag! You probably distributed your existing t-shirts and stickers during another event. Make your hackathon memorable and go for new, exciting, and exclusive designs. Shirts are great, and hoodies reign supreme. But think about cool swag participants may not have already. Think of something that could become their new staple item, like backup batteries or hats (both popular at HackerOne). Personally, my own home features some towels and slippers from hackathons!

Our favorite resources about open source Git cheat sheet Advanced Linux commands cheat sheet Open source alternatives Free online course: RHEL technical overview Check out more cheat sheets 9. Get the word out

Setting your goals and deciding on amazing grand prizes and swag are all important steps. But how will anyone know your hackathon is happening if you don't get the word out? You need to investigate the available channels carefully, and you need to be bold with your promotion. I'm talking blogs, vlogs, emails, social media—anything you can get your hands on.

However, depending on your defined goals, you need to invest in the appropriate channel. Where you advertise depends on who you want to invite to your hackathon.

IIf you want to attract more experienced users, target big organizations where your project is used. LinkedIn and email promotion would be most effective here.
If you want to bring in new and less experienced users, you're better off targeting universities and boot camps. Promoting the event on community-based media, like Mastodon, Matrix, Mattermost, Reddit, Discourse, Discord, and any place your target audience hangs out is a better choice.

10. Managing the long tail

Yay, the hackathon is over! Now all hackathon-related activities can stop, and we no longer need to pull resources, right? Wrong! Think of hackathons as only one step of the road in a series of events in your software development and community building. To deem your hackathon a success, you must be prepared to engage in post-event activities.

Communicating your results: Don't forget to communicate hackathon outcomes internally and externally. Demonstrate the ownership the community members gained during the hackathon to grow trust in your community and project.
Building community: Lean on your hackathon participants for future community activity.
Putting together the retrospective: What went well, what went terrible, what was meh, what surprised you? This analysis is how you grow, change, and iterate. Don't forget to do a blameless retro as soon as possible so it is all fresh in your mind.

Wrap up

If you started reading this article thinking that hackathons aren't that hard to pull off, I'm sorry to have burst your bubble! And although I sincerely believe hackathons are a great way to engage and communicate with your community on so many levels, having just the intention does not guarantee the results.

For a hackathon to be successful, you need to be meticulous and prepared to invest significant resources and effort to plan and execute it properly.

Thank you for reading, and I hope this checklist helps you successfully organize your next hackathon!

Follow this blueprint for planning, hosting, and managing a successful hackathon.

Image by:

Image by Mapbox Uncharted ERG, CC-BY 3.0 US

Programming Community management Download this article as a PDF and EPUB A 10-step guide for a successful hackathon This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Pages