Open-source News

Google proclaims official Android RISC-V support - comments  GSMArena.com

How to Host A Monero Node on Tor in Linux  Make Tech Easier

Linux software, steps to migrate Windows from big drive to small drive?  HardForum

Endpoint Security Solutions for the Growing Threat Landscape  BlackBerry Blog

Researchers Discover New Linux Malware Targeting WordPress Sites  OODA Loop

In addition to Fedora 38 now allowing "no-omit-frame-pointer" to enhance profiling/debugging with possible performance costs, this next Fedora Linux release is also planning to use "_FORTIFY_SOURCE=3" compiler defenses to further bolster security...

Humble Choice for January 2023 has DOOM Eternal, a good month ...  GamingOnLinux

AlmaLinux, CentOS Stream, Clear Linux, Debian, Fedora & Ubuntu ...  Phoronix

Over the holidays some fun benchmarking was to be had with the dual AMD EPYC 9654 "Genoa" processors providing a combined 192 cores / 384 threads and seeing how various modern Linux distributions were competing for this flagship 4th Gen EPYC server configuration. Up on the testing block was AlmaLinux 9.1, CentOS Stream 9, Clear Linux 37930, Debian 12 Testing, Fedora Server 37, Ubuntu 22.04.1 LTS, Ubuntu 22.10, and Ubuntu 23.04 daily.

Classic 1990s series Catacomb lives on with the CatacombGL ...  GamingOnLinux

Samuel Pitoiset of Valve's Linux graphics team has landed a few patches into Mesa 23.0 for further reducing the CPU overhead of the Vulkan driver's draw path...

AMD Ray Tracing on Linux gets closer with Mesa enabling specific ...  GamingOnLinux

Since GCC 11 there has been support for AMX and the upcoming Sapphire Rapids CPU features, which has been further improved in the open-source compiler over the past two years. GCC 13 meanwhile as the next GNU Compiler Collection release is bringing Meteor Lake and Sierra Forest, Grand Ridge, and Granite Rapids. Basic enablement of Intel's Emerald Rapids meanwhile was merged yesterday for GCC 13 too...

Microsoft released CBL-Mariner 2.0.20221222 on Tuesday as their first update of 2023 for their in-house Linux distribution that is used for a variety of purposes within the company from Azure to other behind-the-scenes Linux OS use...

Microsoft's CBL-Mariner 2.0.20221222 Linux Distro Now Allows ...  Phoronix

Linbit pushes DRDB-based software as SAN replacement – Blocks ...  Blocks and Files

GNOME's Mutter now allows disabling XWayland support at build-time if so desired. This is part of the broader GNOME effort for making X11 support optional and ultimately allowing for a modern Wayland-only environment if so desired and without carrying legacy X11 cruft...

New shc-based Linux Malware Targeting Systems with Cryptocurrency Miner  The Hacker News

Valve Wants to Bring HDR to Linux Systems  GameRant

Customize Apache ShardingSphere high availability with MySQL zhaojinchao Wed, 01/04/2023 - 03:00

Users have many options to customize and extend ShardingSphere's high availability (HA) solutions. Our team has completed two HA plans: A MySQL high availability solution based on MGR and an openGauss database high availability solution contributed by some community committers. The principles of the two solutions are the same.

Below is how and why ShardingSphere can achieve database high availability using MySQL as an example:

Image by:

(Zhao Jinchao, CC BY-SA 4.0)

Prerequisite

ShardingSphere checks if the underlying MySQL cluster environment is ready by executing the following SQL statement. ShardingSphere cannot be started if any of the tests fail.

Check if MGR is installed:

SELECT * FROM information_schema.PLUGINS WHERE PLUGIN_NAME='group_replication'

View the MGR group member number. The underlying MGR cluster should consist of at least three nodes:

SELECT COUNT(*) FROM performance_schema.replication_group_members

Check whether the MGR cluster's group name is consistent with that in the configuration. The group name is the marker of an MGR group, and each group of an MGR cluster only has one group name:

SELECT * FROM performance_schema.global_variables WHERE VARIABLE_NAME='group_replication_group_name' 

Check if the current MGR is set as the single primary mode. Currently, ShardingSphere does not support dual-write or multi-write scenarios. It only supports single-write mode:

SELECT * FROM performance_schema.global_variables WHERE VARIABLE_NAME='group_replication_single_primary_mode'

Query all the node hosts, ports, and states in the MGR group cluster to check if the configured data source is correct:

SELECT MEMBER_HOST, MEMBER_PORT, MEMBER_STATE FROM performance_schema.replication_group_membersDynamic primary database discovery

ShardingSphere finds the primary database URL according to the query master database SQL command provided by MySQL:

private String findPrimaryDataSourceURL(final Map<String, DataSource> dataSourceMap) {
    String RESULT = "";
    String SQL = "SELECT MEMBER_HOST, MEMBER_PORT FROM performance_schema.replication_group_members WHERE MEMBER_ID = "
            + "(SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME = 'group_replication_primary_member')";
    FOR (DataSource each : dataSourceMap.values()) {
        try (Connection connection = each.getConnection();
             Statement statement = connection.createStatement();
             ResultSet resultSet = statement.executeQuery(SQL)) {
            IF (resultSet.next()) {
                RETURN String.format("%s:%s", resultSet.getString("MEMBER_HOST"), resultSet.getString("MEMBER_PORT"));
            }
        } catch (final SQLException ex) {
            log.error("An exception occurred while find primary data source url", ex);
        }
    }
    RETURN RESULT;
}

Compare the primary database URLs found above one by one with the dataSources URLs configured. The matched data source is the primary database. It will be updated to the current ShardingSphere memory and be perpetuated to the registry center, through which it will be distributed to other compute nodes in the cluster.

Image by:

(Zhao Jinchao, CC BY-SA 4.0)

Dynamic secondary database discovery

There are two types of secondary database states in ShardingSphere: enable and disable. The secondary database state will be synchronized to the ShardingSphere memory to ensure that read traffic can be routed correctly.

Get all the nodes in the MGR group:

SELECT MEMBER_HOST, MEMBER_PORT, MEMBER_STATE FROM performance_schema.replication_group_members

Disable secondary databases:

private void determineDisabledDataSource(final String schemaName, final Map<String, DataSource> activeDataSourceMap,
                                         final List<String> memberDataSourceURLs, final Map<String, String> dataSourceURLs) {
    FOR (Entry<String, DataSource> entry : activeDataSourceMap.entrySet()) {
        BOOLEAN disable = TRUE;
        String url = NULL;
        try (Connection connection = entry.getValue().getConnection()) {
            url = connection.getMetaData().getURL();
            FOR (String each : memberDataSourceURLs) {
                IF (NULL != url && url.contains(each)) {
                    disable = FALSE;
                    break;
                }
            }
        } catch (final SQLException ex) {
            log.error("An exception occurred while find data source urls", ex);
        }
        IF (disable) {
            ShardingSphereEventBus.getInstance().post(NEW DataSourceDisabledEvent(schemaName, entry.getKey(), TRUE));
        } ELSE IF (!url.isEmpty()) {
            dataSourceURLs.put(entry.getKey(), url);
        }
    }
}

Whether the secondary database is disabled is based on the data source configured and all the nodes in the MGR group.

ShardingSphere can check one by one whether the data source configured can obtain Connection properly and verify whether the data source URL contains nodes of the MGR group.

If Connection cannot be obtained or the verification fails, ShardingSphere will disable the data source by an event trigger and synchronize it to the registry center.

Enable secondary databases:

private void determineEnabledDataSource(final Map<String, DataSource> dataSourceMap, final String schemaName,
                                        final List<String> memberDataSourceURLs, final Map<String, String> dataSourceURLs) {
    FOR (String each : memberDataSourceURLs) {
        BOOLEAN enable = TRUE;
        FOR (Entry<String, String> entry : dataSourceURLs.entrySet()) {
            IF (entry.getValue().contains(each)) {
                enable = FALSE;
                break;
            }
        }
        IF (!enable) {
            continue;
        }
        FOR (Entry<String, DataSource> entry : dataSourceMap.entrySet()) {
            String url;
            try (Connection connection = entry.getValue().getConnection()) {
                url = connection.getMetaData().getURL();
                IF (NULL != url && url.contains(each)) {
                    ShardingSphereEventBus.getInstance().post(NEW DataSourceDisabledEvent(schemaName, entry.getKey(), FALSE));
                    break;
                }
            } catch (final SQLException ex) {
                log.error("An exception occurred while find enable data source urls", ex);
            }
        }
    }
}

After the crashed secondary database is recovered and added to the MGR group, the configuration will be checked to see whether the recovered data source is used. If yes, the event trigger will tell ShardingSphere that the data source needs to be enabled.

More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources Heartbeat mechanism

The heartbeat mechanism is introduced to the HA module to ensure that the primary-secondary states are synchronized in real-time.

By integrating the ShardingSphere sub-project ElasticJob, the above processes are executed by the ElasticJob scheduler framework in the form of Job when the HA module is initialized, thus achieving the separation of function development and job scheduling.

Even if developers need to extend the HA function, they do not need to care about how jobs are developed and operated:

private void initHeartBeatJobs(final String schemaName, final Map<String, DataSource> dataSourceMap) {
    Optional<ModeScheduleContext> modeScheduleContext = ModeScheduleContextFactory.getInstance().get();
    IF (modeScheduleContext.isPresent()) {
        FOR (Entry<String, DatabaseDiscoveryDataSourceRule> entry : dataSourceRules.entrySet()) {
            Map<String, DataSource> dataSources = dataSourceMap.entrySet().stream().filter(dataSource -> !entry.getValue().getDisabledDataSourceNames().contains(dataSource.getKey()))
                    .collect(Collectors.toMap(Entry::getKey, Entry::getValue));
            CronJob job = NEW CronJob(entry.getValue().getDatabaseDiscoveryType().getType() + "-" + entry.getValue().getGroupName(),
                each -> NEW HeartbeatJob(schemaName, dataSources, entry.getValue().getGroupName(), entry.getValue().getDatabaseDiscoveryType(), entry.getValue().getDisabledDataSourceNames())
                            .execute(NULL), entry.getValue().getHeartbeatProps().getProperty("keep-alive-cron"));
            modeScheduleContext.get().startCronJob(job);
        }
    }
}Wrap up

So far, Apache ShardingSphere's HA feature has proven applicable for MySQL and openGauss HA solutions. Moving forward, it will integrate more MySQL HA products and support more database HA solutions.

As always, if you're interested, you're more than welcome to join us and contribute to the Apache ShardingSphere project.

Apache ShardingSphere Open Source Project Links:

• ShardingSphere GitHub
• ShardingSphere Twitter
• ShardingSphere Slack
• Contributor guidelines

Learn how and why ShardingSphere can achieve database high availability using MySQL as an example.

Image by:

Image by Mapbox Uncharted ERG, CC-BY 3.0 US

Databases What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.