Open-source News

Clément Lefèbvre has published the latest monthly status report for the Linux Mint distribution that is the popular desktop Linux distribution based on Ubuntu Linux...

Linux Mint Makes Improvements Around Flatpaks With Update Manager Integration  Phoronix

Landing in "net-next" on Monday is wired networking support for the MotorComm YT8521 Ethernet Gigabit PHY. This network ASIC may not ring a bell for most folks, but is used so far by one notable RISC-V development board...

MotorComm YT8521 Gigabit Ethernet Support Coming For Linux 6.2  Phoronix

With the fifteen Linux hardware reviews and 245 original open-source/Linux news stories written by your's truly last month, here is a look back at what was exciting Phoronix readers the most from Google's new KataOS to the release of the speedy Python 3.11, Linux 6.1 taking shaping, and Intel releasing Arc Graphics A750 and A770 graphics cards...

Arc Graphics, Google KataOS, Python 3.11 & Linux 6.1 Excited Open-Source Enthusiasts  Phoronix

Adding to the long list of Zink OpenGL-on-Vulkan driver improvements coming with Mesa 22.3 this quarter is now a working Mesa front-end shader caching implementation...

OBS Studio as the leading open-source and cross-platform screencasting/streaming app that is popular with gamers and YouTubers is now out with version 28.1...

Linux world gains ability to repair exFAT drives  The Register

Linux Lite 6.2: Latest release from distro with a misleading name  The Register

Linux Fu: Easy VMs  Hackaday

Linux world gains ability to repair exFAT drives  The Register

How to Force Linux Users to Change Password at Next Login  MUO - MakeUseOf

First Arch Linux ISO Release Powered by Linux Kernel 6.0 Is Now Available for Download  9to5Linux

Get started with Parseable, an open source log storage and observability platform Nitish Tiwari Tue, 11/01/2022 - 03:00

Log data is one of the fastest-growing segments across data storage. It's also one of the most complicated spaces. There are several products and solutions with overlapping use cases and confusing marketing.

This article looks at Parseable, a log storage and observability platform. Parseable is geared towards a better user experience, with an easy-to-deploy and use interface and a simple, cloud-native architecture. I'll also show how to set up Parseable with FluentBit to store logs.

More on Microservices Microservices cheat sheet How to explain microservices to your CEO Free eBook: Microservices vs. service-oriented architecture Free online course: Developing cloud-native applications with microservices arc… Latest microservices articles What is Parseable?

Parseable is a free and open source log storage and observability platform. Written in Rust, Parseable leverages data compression, storage, and networking advances to bring a simple, efficient logging platform that just works.

Some core concepts for building Parseable are:

Indexing free

Traditionally, text search engines like Elastic have doubled as log storage platforms. This makes sense because log data must be searched to be really useful. But indexing comes at a high cost. It is CPU intensive and slows down ingestion. Also, the index data generated by these systems are of the same order of storage as the raw log data. This doubles the storage cost and increases complexity. Parseable changes this. With columnar data formats (parquet), it is possible to compress and query the log data efficiently without indexing it.

Ownership of both data and content

With parquet as the storage format and stored in standard object storage buckets, users own their log data and have complete access to the actual content. This means users can easily use analysis tools like Spark, Presto, or TensorFlow to extract more value from the data. This feature is extremely powerful, opening up new avenues of data analysis.

Fluid schema

Logs are generally semi-structured by nature, and they're ever-evolving. For example, a developer may start with a log schema like this:

{
  "Status": "Ready",
  "Application": "Example"
}

But as more information is collected, the log schema may evolve to:

{
  "Status": "Ready",
  "Application": {
    "UserID": "3187F492-8449-4486-A2A0-015AE34F1D09",
    "Name": "Example"
  }
}

Engineering and SRE teams regularly face schema-related issues. Parseable solves this with a fluid schema approach that lets users change the schema on the fly.

Simple ingestion

The current ingestion mechanism for logging platforms is quite convoluted, with several available protocols and connectors. Parseable aims to make log ingestion as easy as possible. The result is you can use HTTP POST calls to send logs to Parseable. No complicated SDKs are required.

What if you want to use a logging agent like FluentBit, Vector, LogStash, or others? Almost all the major log collectors support HTTP, so Parseable is already compatible with your favorite log collection agent.

Get started

You can use a Docker image to try out Parseable. This image shows Parseable in demo mode, using publicly-accessible object storage.

$ cat << EOF > parseable-env
P_S3_URL=https://minio.parseable.io:9000
P_S3_ACCESS_KEY=minioadmin
P_S3_SECRET_KEY=minioadmin
P_S3_REGION=us-east-1
P_S3_BUCKET=parseable
P_LOCAL_STORAGE=/data
P_USERNAME=parseable
P_PASSWORD=parseable
EOF

$ mkdir -p /tmp/data

$ docker run \
  -p 8000:8000 \
  --env-file parseable-env \
  -v /tmp/data:/data \
  parseable/parseable:latest

Log in to the Parseable UI using the credentials passed here (that's parseable and parseable.) The demo already contains some data because Parseable is pointing to the publicly-open bucket.

Make sure to change the bucket and credentials to your object storage instance before sending any data to Parseable.

Refer to the documentation to understand how Parseable works and how to ingest logs.

Set up FluentBit to send logs to Parseable

You can use a Docker compose file to configure both Parseable and FluentBit, making it easier to set up and tear down as needed.

First, save this file as fluent-bit.conf in a directory. The file is the configuration used to send data to Parseable.

[SERVICE]
  Flush 5
  Daemon Off
  Log_Level debug

[INPUT]
  Name dummy
  Tag dummy

[OUTPUT]
  Name http
  Match *
  Host parseable
  http_User parseable
  http_Passwd parseable
  format json
  Port 8000
  Header X-P-META-meta1 value1
  Header X-P-TAG-tag1 value1
  URI /api/v1/logstream/fluentbit1
  Json_date_key timestamp
  Json_date_format iso8601

Now save the following file as docker-compose.yaml in the same directory as above:

version: "3.7"

services:
  fluent-bit:
    image: fluent/fluent-bit
    volumes:
     - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    depends_on:
     - parseable

  parseable:
    image: parseable/parseable
    ports:
     - "8000:8000"
    environment:
     - P_S3_URL=https://minio.parseable.io:9000
      - P_S3_ACCESS_KEY=minioadmin
      - P_S3_SECRET_KEY=minioadmin
      - P_S3_REGION=us-east-1
      - P_S3_BUCKET=parseable
      - P_LOCAL_STORAGE=/tmp/data
      - P_USERNAME=parseable
      - P_PASSWORD=parseable

The docker-compose.yaml refers to the fluent-bit.conf file and passes it to the FluentBit container as the configuration file.

Parseable is deployed with the default configuration (as in the above Docker setup). You can observe the data FluentBit container sent to Parseable in the Parseable Console running at http://localhost:8000.

Wrap up

In this article, you've taken your first look at Parseable, the open source log storage and analysis platform built in Rust. A single Docker command gets you started with Parseable so you can experience the UI and establish FluentBit as a data source. If you think this looks too easy, then it's probably time to try Parseable!

Written in Rust, Parseable leverages data compression, storage, and networking advances to bring a simple, efficient logging platform that just works.

Image by:

Opensource.com

Rust Microservices Cloud What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Transfer files and folders from Windows to Linux with WinSCP Paul Tue, 11/01/2022 - 03:00

Sometimes you need to transfer files over a network. There are lots of file sharing services out there, but most require that you send your file to the Internet. This seems like a long way to go (not to mention the privacy concerns) when two computers are right beside each other, or at least in the same building. The open source WinSCP utility makes it quick and easy to transfer a file or a folder of files over the network from your Windows computer to your Linux computer.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles IP address

Before you can make the transfer, you must know the IP address or fully-qualified domain name of the destination computer. Assuming it's a computer on your same network, and that you're not running a DNS server to resolve computer names, you can find the destination IP address using the ip command on the Linux machine:

[linux]$ ip addr show | grep 'inet '  
inet 127.0.0.1/8 scope host lo  
inet 192.168.1.23/24 brd 10.0.1.255 scope global noprefixroute eth0  

In all cases, 127.0.0.1 is a loopback address that the computer uses only to talk to itself, so in this example the correct address is 192.168.1.23. On your system, the IP address is likely to be different. If you're not sure which is which, you can try each one in succession until you get the right one (and then write it down somewhere!)

Alternatively, you can look in your router's settings, which list all addresses assigned over DHCP.

Firewalls and servers

The WinSCP command uses the OpenSSH protocol, so your Linux computer must be running the OpenSSH server software, and its firewall must allow SSH traffic.

If you're not sure whether your Linux machine is running SSH, then run this command on the Linux machine:

[linux]$ sudo systemctl enable --now sshd

To ensure your firewall allows SSH traffic, run this command:

[linux]$ sudo firewall-cmd --add-service ssh --permanent

For more information on firewalls on Linux, read Make Linux stronger with firewalls.

Using WinSCP

WinSCP is an open source SSH file transfer application for Microsoft Windows. To use it, you first must download and install it.

Once you're installed it, open WinSCP and select the SCP option in the File Protocol field.

Add the IP address or DNS name of your Linux computer in the Host name field, and enter 22 in the Port number field. Enter you user name and password for the Linux computer, and then click the Login button at the bottom of the WinSCP window.

Image by:

(Paul Laubscher, CC BY-SA 4.0)

Verify that you are authenticated to the Linux computer. Upon success, your Linux computer's IP address or DNS name appears at the top of the window.

Image by:

(Paul Laubscher, CC BY-SA 4.0)

 

Now you can drag and drop a file (I used winscp-test.txt as an example) from the left Windows pane to the destination Linux computer pane on the right, and the file transfers.

Image by:

(Paul Laubscher, CC BY-SA 4.0)

Alternatively, you can right-click on a file in the left pane and upload it to the remote destination in the right pane.

Image by:

(Paul Laubscher, CC BY-SA 4.0)

Verify the copy

Open a Linux terminal and use the ls command to view the transferred winscp-test.txt file. In my example, it appears in my home directory, /_home_/sysadmin.

$ ls
Desktop
Documents
Downloads
Music
Pictures
pscp-test.txt
[...]

You've successfully transferred a file from a Windows computer to a Linux computer over the network!

Of course, you can use the same technique as above to transfer files and folders from a Linux computer to a Windows computer.

Remote copying

With the power of the open source WinSCP application, you have access to any computer in your house or workplace, to servers you have accounts on, and even mobile, edge, and Internet of Things devices. Use this great tool to transfer files as easily as you would copy a file from one local directory to another!

If you're looking for a way to quickly transfer files from your Windows computer to your Linux computer, then the open source WinSCP utility makes it easy to transfer a file or a folder of files over the network.

Image by:

Opensource.com

Linux Windows What to read next Transfer files and folders from Windows to Linux with PSCP This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Linux Lite 6.2 Released  It's FOSS News

OBS Studio 28.1 Released with Updated NVENC Presets, Virtual Camera Improvements  9to5Linux

Attackers searching for more bugs, using documents less and exfiltrating more data  SC Media

Attackers searching for more bugs, using documents less and exfiltrating more data  SC Media