Introduction🔗
This page walks you through the release process of the Iceberg project. Here you can read about the release process in general for an Apache project.
Decisions about releases are made by three groups:
- Release Manager: Does the work of creating the release, signing it, counting votes, announcing the release and so on. Requires the assistance of a committer for some steps.
- The community: Performs the discussion of whether it is the right time to create a release and what that release should contain. The community can also cast non-binding votes on the release.
- PMC: Gives binding votes on the release.
This page describes the procedures that the release manager and voting PMC members take during the release process.
Setup🔗
To create a release candidate, you will need:
- Apache LDAP credentials for Nexus and SVN
- A GPG key for signing, published in KEYS
If you have not published your GPG key yet, you must publish it before sending the vote email by doing:
svn co https://dist.apache.org/repos/dist/dev/iceberg icebergsvn
cd icebergsvn
echo "" >> KEYS # append a newline
gpg --list-sigs <YOUR KEY ID HERE> >> KEYS # append signatures
gpg --armor --export <YOUR KEY ID HERE> >> KEYS # append public key block
svn commit -m "add key for <YOUR NAME HERE>"
Nexus access🔗
Nexus credentials are configured in your personal ~/.gradle/gradle.properties
file using mavenUser
and mavenPassword
:
PGP signing🔗
The release scripts use the command-line gpg
utility so that signing can use the gpg-agent and does not require writing your private key's passphrase to a configuration file.
To configure gradle to sign convenience binary artifacts, add the following settings to ~/.gradle/gradle.properties
:
To use gpg
instead of gpg2
, also set signing.gnupg.executable=gpg
For more information, see the Gradle signing documentation.
Apache repository🔗
The release should be executed against https://github.com/apache/iceberg.git
instead of any fork.
Set it as remote with name apache
for release if it is not already set up.
Creating a release candidate🔗
Initiate a discussion about the release with the community🔗
This step can be useful to gather ongoing patches that the community thinks should be in the upcoming release.
The communication can be started via a [DISCUSS] mail on the dev@ channel and the desired tickets can be added to the github milestone of the next release.
Note, creating a milestone in github requires a committer. However, a non-committer can assign tasks to a milestone if added to the list of collaborators in .asf.yaml
The release status is discussed during each community sync meeting. Release manager should join the meeting to report status and discuss any release blocker.
Build the source release🔗
To create the source release artifacts, run the source-release.sh
script with the release version and release candidate number:
Example console output:
Preparing source for apache-iceberg-0.13.0-rc1
Adding version.txt and tagging release...
[main ca8bb7d0] Add version.txt for release 0.13.0
1 file changed, 1 insertion(+)
create mode 100644 version.txt
Pushing apache-iceberg-0.13.0-rc1 to origin...
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 12 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 433 bytes | 433.00 KiB/s, done.
Total 4 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To https://github.com/apache/iceberg.git
* [new tag] apache-iceberg-0.13.0-rc1 -> apache-iceberg-0.13.0-rc1
Creating tarball using commit ca8bb7d0821f35bbcfa79a39841be8fb630ac3e5
Signing the tarball...
Checking out Iceberg RC subversion repo...
Checked out revision 52260.
Adding tarball to the Iceberg distribution Subversion repo...
A tmp/apache-iceberg-0.13.0-rc1
A tmp/apache-iceberg-0.13.0-rc1/apache-iceberg-0.13.0.tar.gz.asc
A (bin) tmp/apache-iceberg-0.13.0-rc1/apache-iceberg-0.13.0.tar.gz
A tmp/apache-iceberg-0.13.0-rc1/apache-iceberg-0.13.0.tar.gz.sha512
Adding tmp/apache-iceberg-0.13.0-rc1
Adding (bin) tmp/apache-iceberg-0.13.0-rc1/apache-iceberg-0.13.0.tar.gz
Adding tmp/apache-iceberg-0.13.0-rc1/apache-iceberg-0.13.0.tar.gz.asc
Adding tmp/apache-iceberg-0.13.0-rc1/apache-iceberg-0.13.0.tar.gz.sha512
Transmitting file data ...done
Committing transaction...
Committed revision 52261.
Creating release-announcement-email.txt...
Success! The release candidate is available here:
https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-0.13.0-rc1
Commit SHA1: ca8bb7d0821f35bbcfa79a39841be8fb630ac3e5
We have generated a release announcement email for you here:
/Users/jackye/iceberg/release-announcement-email.txt
Please note that you must update the Nexus repository URL
contained in the mail before sending it out.
The source release script will create a candidate tag based on the HEAD revision in git and will prepare the release tarball, signature, and checksum files. It will also upload the source artifacts to SVN.
Note the commit SHA1 and candidate location because those will be added to the vote thread.
Once the source release is ready, use it to stage convenience binary artifacts in Nexus.
Build and stage convenience binaries🔗
Convenience binaries are created using the source release tarball from in the last step.
Untar the source release and go into the release directory:
To build and publish the convenience binaries, run the dev/stage-binaries.sh
script. This will push to a release staging repository.
Disable gradle parallelism by setting org.gradle.parallel=false
in gradle.properties
.
Next, you need to close the staging repository:
- Go to Nexus and log in
- In the menu on the left, choose "Staging Repositories"
- Select the Iceberg repository
- If multiple staging repositories are created after running the script, verify that gradle parallelism is disabled and try again.
- At the top, select "Close" and follow the instructions
- In the comment field use "Apache Iceberg <version> RC<num>"
Start a VOTE thread🔗
The last step for a candidate is to create a VOTE thread on the dev mailing list.
The email template is already generated in release-announcement-email.txt
with some details filled.
Example title subject:
Example content:
Hi everyone,
I propose the following RC to be released as official Apache Iceberg <VERSION> release.
The commit id is <SHA1>
* This corresponds to the tag: apache-iceberg-<VERSION>-rc<NUM>
* https://github.com/apache/iceberg/commits/apache-iceberg-<VERSION>-rc<NUM>
* https://github.com/apache/iceberg/tree/<SHA1>
The release tarball, signature, and checksums are here:
* https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-<VERSION>-rc<NUM>/
You can find the KEYS file here:
* https://dist.apache.org/repos/dist/dev/iceberg/KEYS
Convenience binary artifacts are staged in Nexus. The Maven repository URL is:
* https://repository.apache.org/content/repositories/orgapacheiceberg-<ID>/
This release includes important changes that I should have summarized here, but I'm lazy.
Please download, verify, and test.
Please vote in the next 72 hours. (Weekends excluded)
[ ] +1 Release this as Apache Iceberg <VERSION>
[ ] +0
[ ] -1 Do not release this because...
Only PMC members have binding votes, but other community members are encouraged to cast
non-binding votes. This vote will pass if there are 3 binding +1 votes and more binding
+1 votes than -1 votes.
When a candidate is passed or rejected, reply with the voting result:
Thanks everyone who participated in the vote for Release Apache Iceberg <VERSION> RC<NUM>.
The vote result is:
+1: 3 (binding), 5 (non-binding)
+0: 0 (binding), 0 (non-binding)
-1: 0 (binding), 0 (non-binding)
Therefore, the release candidate is passed/rejected.
Finishing the release🔗
After the release vote has passed, you need to release the last candidate's artifacts.
But note that releasing the artifacts should happen around the same time the new docs are released so make sure the documentation changes are prepared when going through the below steps.
Publishing the release🔗
First, copy the source release directory to releases:
svn mv https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-<VERSION>-rcN https://dist.apache.org/repos/dist/release/iceberg/apache-iceberg-<VERSION> -m "Iceberg: Add release <VERSION>"
Note
The above step requires PMC privileges to execute.
Next, add a release tag to the git repository based on the passing candidate tag:
git tag -am 'Release Apache Iceberg <VERSION>' apache-iceberg-<VERSION> apache-iceberg-<VERSION>-rcN
Then release the candidate repository in Nexus.
Announcing the release🔗
To announce the release, wait until Maven central has mirrored the Apache binaries, then update the Iceberg site and send an announcement email:
I'm pleased to announce the release of Apache Iceberg <VERSION>!
Apache Iceberg is an open table format for huge analytic datasets. Iceberg
delivers high query performance for tables with tens of petabytes of data,
along with atomic commits, concurrent writes, and SQL-compatible table
evolution.
This release can be downloaded from: https://www.apache.org/dyn/closer.cgi/iceberg/<TARBALL NAME WITHOUT .tar.gz>/<TARBALL NAME>
Release notes: https://iceberg.apache.org/releases/#XYZ-release
Java artifacts are available from Maven Central.
Thanks to everyone for contributing!
Update revapi🔗
Create a PR in the iceberg
repo to make revapi run on the new release. For an example see this PR.
Update GitHub🔗
- Create a PR in the
iceberg
repo to add the new version to the github issue template. For an example see this PR. - Draft a new release to update Github to show the latest release. A changelog can be generated automatically using Github.
Update DOAP (ASF Project Description)🔗
- Create a PR to update the release version in doap.rdf file, in the
<release/>
section:
<release>
<Version>
<name>x.y.z</name>
<created>yyyy-mm-dd</created>
<revision>x.y.z</revision>
</Version>
</release>
Documentation Release🔗
Please follow the instructions on the GitHub repository in the README.md
in the site/
directory.
How to Verify a Release🔗
Each Apache Iceberg release is validated by the community by holding a vote. A community release manager will prepare a release candidate and call a vote on the Iceberg dev list. To validate the release candidate, community members will test it out in their downstream projects and environments. It's recommended to report the Java, Scala, Spark, Flink and Hive versions you have tested against when you vote.
In addition to testing in downstream projects, community members also check the release's signatures, checksums, and license documentation.
Validating a source release candidate🔗
Release announcements include links to the following:
- A source tarball
- A signature (.asc)
- A checksum (.sha512)
- KEYS file
- GitHub change comparison
After downloading the source tarball, signature, checksum, and KEYS file, here are instructions on how to verify signatures, checksums, and documentation.
Verifying Signatures🔗
First, import the keys.
Next, verify the .asc
file.
Verifying Checksums🔗
Verifying License Documentation🔗
Untar the archive and change into the source directory.
Run RAT checks to validate license headers.
Verifying Build and Test🔗
To verify that the release candidate builds properly, run the following command.
Testing release binaries🔗
Release announcements will also include a maven repository location. You can use this location to test downstream dependencies by adding it to your maven or gradle build.
To use the release in your maven build, add the following to your POM
or settings.xml
:
...
<repositories>
<repository>
<id>iceberg-release-candidate</id>
<name>Iceberg Release Candidate</name>
<url>${MAVEN_URL}</url>
</repository>
</repositories>
...
To use the release in your gradle build, add the following to your build.gradle
:
Note
Replace ${MAVEN_URL}
with the URL provided in the release announcement
Verifying with Spark🔗
To verify using spark, start a spark-shell
with a command like the following command (use the appropriate
spark-runtime jar for the Spark installation):
spark-shell \
--conf spark.jars.repositories=${MAVEN_URL} \
--packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.0 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.local.type=hadoop \
--conf spark.sql.catalog.local.warehouse=$PWD/warehouse \
--conf spark.sql.catalog.local.default-namespace=default \
--conf spark.sql.defaultCatalog=local
Verifying with Flink🔗
To verify using Flink, start a Flink SQL Client with the following command:
wget ${MAVEN_URL}/iceberg-flink-runtime-1.20/1.7.0/iceberg-flink-runtime-1.20-1.7.0.jar
sql-client.sh embedded \
-j iceberg-flink-runtime-1.20-1.7.0.jar \
-j flink-connector-hive_2.12-1.20.jar \
shell
Voting🔗
Votes are cast by replying to the release candidate announcement email on the dev mailing list
with either +1
, 0
, or -1
.
[ ] +1 Release this as Apache Iceberg 1.7.0 [ ] +0 [ ] -1 Do not release this because...
In addition to your vote, it's customary to specify if your vote is binding or non-binding. Only members of the Project Management Committee have formally binding votes. If you're unsure, you can specify that your vote is non-binding. To read more about voting in the Apache framework, checkout the Voting information page on the Apache foundation's website.