summaryrefslogtreecommitdiffstats
path: root/tools/gn/toolchain_manager.cc
Commit message (Collapse)AuthorAgeFilesLines
* Refactor the GN threading model to reduce locking and simplify execution.brettw@chromium.org2013-11-081-593/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | Old design: item_tree manages dependency resolution, target_manager creates and hooks up the target dependencies, and the toolchain manager does all kinds of crazy ordering and loading stuff on every thread. There is lots of locking every time the dependency tree is modified. This patch completely deletes the toolchain manager, item tree, item node, and target manager. The new design is that the build files are executed on background threads, but then they come back to the main thread which assembles the dependency tree and schedules other file loads. This eliminates almost all locking and a lot of complexity. The Loader now manages loading the different build files in the correct context, and managing dependencies (loading the build config first, for example). The Builder manages the dependency tree and requests that the Loader load files that it encounters references to. This simpler design reduces non-test code by ~350 lines. In the current gn build, lock acquisitions go down from 1808 to 116 and it saves about 20ms wall clock time (8% faster). This is a bit deceiving, though, because most of the time is spent on pkg-config which is constant. It speeds up running individual build files by 1000-1500%. The work of putting the tree together that used to take up this time inside locks is now transferred to the main thread, which looks like it is *sometimes* a bottleneck. This should be easier to optimize in the future, though. Other tweaks: - I fixed cycle finding - Directories look prettier on Windows using the "desc" command - Tracing output now includes the main thread and marks it as such - I updated the base BUILD.gn file for the more recent tree. R=scottmg@chromium.org Review URL: https://codereview.chromium.org/56433003 git-svn-id: svn://svn.chromium.org/chrome/trunk/src@234032 0039d316-1c4b-4281-b951-d872f2087c98
* GN: toolchain threading cleanupbrettw@chromium.org2013-11-041-21/+28
| | | | | | | | | | | | | | | | | | Remove the thread-unsafe toolchain pointer on the otherwise-threadsafe Settings object. I replaced it with the toolchain label, and moved the is_default flag from the toolchain to the Settings object. This required that I pass the toolchain around in a few more places, but also simplifies some other cases. I removed the toolchain prefix from Ninja rules for the default toolchain since that's not necessary any more for GYP compat. This fixes an annoying double-free in the toolchain manager. I think my current refactor will clean this up in a later phase. R=scottmg@chromium.org Committed: https://src.chromium.org/viewvc/chrome?view=rev&revision=232657 Review URL: https://codereview.chromium.org/51693002 git-svn-id: svn://svn.chromium.org/chrome/trunk/src@232755 0039d316-1c4b-4281-b951-d872f2087c98
* Revert 232657 "GN: toolchain threading cleanup"thakis@chromium.org2013-11-031-28/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Broke linux clang bots: ../../tools/gn/ninja_helper_unittest.cc:20:9:error: field 'settings' will be initialized after field 'toolchain' [-Werror,-Wreorder] This happened on your try jobs too. > GN: toolchain threading cleanup > > Remove the thread-unsafe toolchain pointer on the otherwise-threadsafe Settings object. I replaced it with the toolchain label, and moved the is_default flag from the toolchain to the Settings object. > > This required that I pass the toolchain around in a few more places, but also simplifies some other cases. > > I removed the toolchain prefix from Ninja rules for the default toolchain since that's not necessary any more for GYP compat. > > This fixes an annoying double-free in the toolchain manager. I think my current refactor will clean this up in a later phase. > > R=scottmg@chromium.org > > Review URL: https://codereview.chromium.org/51693002 TBR=brettw@chromium.org Review URL: https://codereview.chromium.org/46313006 git-svn-id: svn://svn.chromium.org/chrome/trunk/src@232661 0039d316-1c4b-4281-b951-d872f2087c98
* GN: toolchain threading cleanupbrettw@chromium.org2013-11-031-21/+28
| | | | | | | | | | | | | | | | Remove the thread-unsafe toolchain pointer on the otherwise-threadsafe Settings object. I replaced it with the toolchain label, and moved the is_default flag from the toolchain to the Settings object. This required that I pass the toolchain around in a few more places, but also simplifies some other cases. I removed the toolchain prefix from Ninja rules for the default toolchain since that's not necessary any more for GYP compat. This fixes an annoying double-free in the toolchain manager. I think my current refactor will clean this up in a later phase. R=scottmg@chromium.org Review URL: https://codereview.chromium.org/51693002 git-svn-id: svn://svn.chromium.org/chrome/trunk/src@232657 0039d316-1c4b-4281-b951-d872f2087c98
* Add tracing and timing info to GNbrettw@chromium.org2013-09-231-0/+12
| | | | | | | | | | | | | This allows outputting a performance summary of time blame, as well as writing a chrome trace log. Moves the help printing functions from command_help to standard_out since I re-used the coloring routines for timing output. BUG= R=scottmg@chromium.org Review URL: https://codereview.chromium.org/24331007 git-svn-id: svn://svn.chromium.org/chrome/trunk/src@224749 0039d316-1c4b-4281-b951-d872f2087c98
* GN: Add some variables, work on net compiling.brettw@chromium.org2013-08-231-1/+1
| | | | | | | | | | | This also adds unit tests for the built-in default variables and fixes a bug. BUG= R=scottmg@chromium.org Review URL: https://codereview.chromium.org/23347003 git-svn-id: svn://svn.chromium.org/chrome/trunk/src@219331 0039d316-1c4b-4281-b951-d872f2087c98
* This make the GN current dir be set on thte scope. This fixes a bug where a ↵brettw@chromium.org2013-08-231-0/+2
| | | | | | | | | | | | | | | | | | | | | | | template execution uses the directory of the file containing the template declaration. This makes it impossible to refer to the input files which are normally specified relative to the invoking file. I added a defined() function that returns true if a variable has been defined. This needed some magic function invocation help because we don't actually want to evaluate the argument. I added an optional second argument to assert for an error message. We now tolerate absolute file paths for sources and outputs. Grit was outputtings these, which was confusiong some of our assertions. We'll now check if such locations are inside our source tree and fix them up accordingly. I fixed a bug in the "desc" command where a target would be listed as a dependent of itself. I also added the target type to the output. Note that I didn't implement MakeAbsolutePathRelativeIfPossible on Windows. I will do this on Windows in a second pass. Currently it's an #error. There are several fixes to custom script target outputs and fixes to variables leftover from when I converted paths to start with two slashes. BUG= R=scottmg@chromium.org Review URL: https://codereview.chromium.org/23396002 git-svn-id: svn://svn.chromium.org/chrome/trunk/src@219161 0039d316-1c4b-4281-b951-d872f2087c98
* This hooks up build flags from the command-line and the toolchain (for ↵brettw@chromium.org2013-08-191-84/+159
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | re-invoking the build). Using this, I was able to make the NaCl build of base compile (if the correct NaCl toolchain is extracted in advance to the right location). This is the first time I actually ran a cross-compile, and Ninja complained about duplicate rule definitions, so I now prefix rules in the non-default toolchain with the toolchain name. There were also some files that went in the wrong directory. I moved the toolchain definitions from the other platforms into a new directory tree in build/toolchains to be more clear. To make the toolchain args get passed to the build configuration, I had to add another state to the toolchain_manager's state machine, since we need to see those flags before starting that part of the build. This is described in more detail at the top of the .cc file. Since I was moving stuff around, I wanted to find references to the thing I was moving, so wrote a new command to find places that reference a given target "gn refs". I added new help for patterns since I used these for the new "refs" command. I removed the requirement that declare_args only be called once in a given scope. It now can be used anywhere to facilitate module-specific arguments. Because build arguments can now be declared anywhere, I added a "gn args" command-line command that will find and print all of the command line arguments passed into the build. It also prints the comment above the argument as a way to document it. The code to extract the comment is a but unfortunate. I think if/when we write an autoformatter, we'll need to incorporate comments into the parse tree more and this can be simplified greatly. Fixes up build.ninja dependencies if you remove a buildfile. Because I specified all input buildfiles in build.ninja, it would fail if you removed any of them. I now write out the build deps (for regenerating the build) in a .d file which will tolerate missing files. BUG= R=scottmg@chromium.org Review URL: https://codereview.chromium.org/22998003 git-svn-id: svn://svn.chromium.org/chrome/trunk/src@218320 0039d316-1c4b-4281-b951-d872f2087c98
* Add support for data deps.brettw@chromium.org2013-08-061-9/+0
| | | | | | | | | | | | | | | | | | | | | | | Data deps are non-linked dependencies of a target. They are built in parallel (they are not input dependencies). I redefined "data" to mean data file dependencies, and added a new "datadeps" contept for non-linked target dependencies. Fix a bug to make it not crash if there's nothing to generate. Add variable documentation for some vars. Removed support for some builtin vars "root output dir name" and related. These had changed definition from when I originally wrote them, and I don't think there's any use for these values. We can add them back if we need. I moved the variable name constant declarations from scope_per_file_provider to the new variables file which includes documentation. I added support for getting the name of the current toolchain via a builtin variable. I removed support for solink_module which is not necessary. This was a way to express a .dll target that isn't linked to its dependees, but that's no longer necessary for datadeps. BUG= R=scottmg@chromium.org Review URL: https://codereview.chromium.org/22290010 git-svn-id: svn://svn.chromium.org/chrome/trunk/src@215976 0039d316-1c4b-4281-b951-d872f2087c98
* Hook up better describing, work on Mac build.brettw@chromium.org2013-08-051-4/+17
| | | | | | | | | | | | | | | | | | | | | This expands the "desc" command to list includes, defined, and flags. It will now optionally annotate the source of the configs producting this list. I folded the old "tree" and "deps" subcommands into the new expanded desc command. This works on the Mac build. Some more Mac flags are set in the environment and I added Mac extensions to some of the lists. It does not yet completely work, however. This adds a lot of Mac/GCC stuff to the current build. I tracked back the current set of defines that the GYP build sets, and set them in the GN build. BUG= R=scottmg@chromium.org Review URL: https://codereview.chromium.org/21636002 git-svn-id: svn://svn.chromium.org/chrome/trunk/src@215722 0039d316-1c4b-4281-b951-d872f2087c98
* Support for non-greedy mode.brettw@chromium.org2013-08-021-4/+17
| | | | | | | | | | | | | | | | | | | | | | | Previously when we encountered a target declaration, we would always recursively load the deps and write the build file for that target. This is "greedy mode." This patch adds a non-greedy mode where a "should generate" but is pushed recursively through the dependency tree. In this mode, targets without the bit set will not have their dependencies loaded, and will not have buildfiles written to disk. The current implementation sets greedy mode for the default toolchain, and non-greed mode for alternate toolchains. The reasoning is that because a target depends on something in another toolchain, we don't want to do an entire build in that alternate toolchain, but just do the minimum dependencies in that alternate toolchain. R=scottmg@chromium.org Review URL: https://codereview.chromium.org/21133014 git-svn-id: svn://svn.chromium.org/chrome/trunk/src@215319 0039d316-1c4b-4281-b951-d872f2087c98
* Revert 214325 "Revert 214254 "Add initial prototype for the GN m..."bauerb@chromium.org2013-07-301-0/+480
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The issue was already fixed :) > Revert 214254 "Add initial prototype for the GN meta-buildsystem." > > It broke the check_licenses step on Android (see http://build.chromium.org/p/chromium.linux/builders/Android%20Builder%20%28dbg%29/builds/39904/steps/check_licenses/logs/stdio): > > @@@BUILD_STEP check_licenses@@@ > > /b/build/slave/Android_Builder__dbg_/build/src/android_webview/tools/webview_licenses.py scan > Got LicenseError "missing README.chromium or licenses.py SPECIAL_CASES entry" while scanning tools/gn/secondary/base/third_party/dynamic_annotations > Got LicenseError "missing README.chromium or licenses.py SPECIAL_CASES entry" while scanning tools/gn/secondary/third_party/modp_b64 > < /b/build/slave/Android_Builder__dbg_/build/src/android_webview/tools/webview_licenses.py scan > ERROR: process exited with code 2 > @@@STEP_FAILURE@@@ > > > > Add initial prototype for the GN meta-buildsystem. > > > > This is currently not hooked into the build. To build, add a reference to the > > gn.gyp file to build/all.gyp > > > > R=darin@chromium.org, scottmg@chromium.org > > > > Review URL: https://codereview.chromium.org/21114002 > > TBR=brettw@chromium.org > > Review URL: https://codereview.chromium.org/21084010 TBR=bauerb@chromium.org Review URL: https://codereview.chromium.org/21204003 git-svn-id: svn://svn.chromium.org/chrome/trunk/src@214333 0039d316-1c4b-4281-b951-d872f2087c98
* Revert 214254 "Add initial prototype for the GN meta-buildsystem."bauerb@chromium.org2013-07-301-480/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | It broke the check_licenses step on Android (see http://build.chromium.org/p/chromium.linux/builders/Android%20Builder%20%28dbg%29/builds/39904/steps/check_licenses/logs/stdio): @@@BUILD_STEP check_licenses@@@ > /b/build/slave/Android_Builder__dbg_/build/src/android_webview/tools/webview_licenses.py scan Got LicenseError "missing README.chromium or licenses.py SPECIAL_CASES entry" while scanning tools/gn/secondary/base/third_party/dynamic_annotations Got LicenseError "missing README.chromium or licenses.py SPECIAL_CASES entry" while scanning tools/gn/secondary/third_party/modp_b64 < /b/build/slave/Android_Builder__dbg_/build/src/android_webview/tools/webview_licenses.py scan ERROR: process exited with code 2 @@@STEP_FAILURE@@@ > Add initial prototype for the GN meta-buildsystem. > > This is currently not hooked into the build. To build, add a reference to the > gn.gyp file to build/all.gyp > > R=darin@chromium.org, scottmg@chromium.org > > Review URL: https://codereview.chromium.org/21114002 TBR=brettw@chromium.org Review URL: https://codereview.chromium.org/21084010 git-svn-id: svn://svn.chromium.org/chrome/trunk/src@214325 0039d316-1c4b-4281-b951-d872f2087c98
* Add initial prototype for the GN meta-buildsystem.brettw@chromium.org2013-07-291-0/+480
This is currently not hooked into the build. To build, add a reference to the gn.gyp file to build/all.gyp R=darin@chromium.org, scottmg@chromium.org Review URL: https://codereview.chromium.org/21114002 git-svn-id: svn://svn.chromium.org/chrome/trunk/src@214254 0039d316-1c4b-4281-b951-d872f2087c98
generated by cgit v1.1 at 2025-01-30 17:32:57 +0000