What is the motivation behind axiomatically forcing the underpinning group of a ring to be abelian? Noncommutative rings are vastly more complex than commutative ones, so I am assuming that allowing the additive operation to be noncommuting would just make matters worse. Is there something deeper here, or is this a restriction for the sake of convenience and simplicity?
Yes, there is a deeper reason, at least in my opinion. Most abstract axiomatic constructions in mathematics are inspired by or even equivalent to concrete examples. One of the most basic algebraic objects one can think of is the set SS of mappings from a set S into itself, which has a natural associative multiplication (composition) and a unit (the identity map). Taking these properties as abstract axioms, one obtains the definition of a unital monoid. However, an element m∈M of an abstractly defined unital monoid M can be viewed as the map [lm:M→M]∈MM given by lm(x)=mx. Mapping m∈M to lm∈MM induces an injective homomorphism of monoids so the abstract definition is really just language (albeit quite helpful) since M is isomorphic to a submonoid of a mapping monoid. One can also consider submonoids of SS which contain only bijective maps. Again, taking this additional property (i.e. existence of inverses) as another axiom, one obtains the definition of a group – but every group can be viewed as a subgroup of a group of bijective maps on a set, so the abstract definition is really just language since it does not provide you with anything that truly generalizes the guiding example.
Now for unital rings the story is essentially the same, but whereas for unital monoids the basic model is SS and its submonoids, the basic model for rings is End(G) and its subrings, where G is a abelian group. End(G) is the submonoid of the mapping monoid GG which contains only those maps which are group homomorphisms. The restriction to abelian groups is necessary since the pointwise product (φ,ψ)↦[x↦φ(x)ψ(x)] (which is the addition in the ring End(G)) of endomorphisms is another endomorphism only if G is assumed to be abelian, in general. As with unital monoids and groups, an abstract unital ring R embeds homomorphically and injectively into its own endomorphism ring End(R) by multiplication operators, and this is a homomorphism of unital rings (not just of unital monoids).
Thus, generalizing the definition of a ring in a manner which is suggested by the question is tantamount to generalizing the example End(G) to nonabelian G. As for how this could be done, there are two possibilities (actually more… see below in the edit). First, one could attempt to define a group structure on End(G) other than the pointwise product. This seems somewhat unnatural since the whole point of using End(G) is to exploit the presence of the group product in the first place. Still, one might speculate that for certain nonabelian groups there are structures on End(G) which are somehow derived from the group product and interact well with the composition – such a structure would be quite bizarre since at the very least, the distributive property will be lost, as explained in the other answers. The other avenue of generalization is to continue using the pointwise product, which as you may have noticed makes the entire mapping monoid GG into a group. Thus, it is possible to consider subsets of GG which are closed under both operations: pointwise product and composition. This is the guiding example of a “near ring” as described on Wikipedia. In general, a submonoid M⊂GG such that M⊂End(G)⊂GG can be one of these only when G is abelian, and these examples are the traditionally defined rings.
EDIT 12/23/2013 Earlier today I realized that there is an easy way to create an example of a “ring with nonabelian underlying group”: just take your favorite nonabelian group G written with + (which is horrible, I know) and impose a second law of composition ∗ given by (x,y)↦x∗y=eG. It is easily verified that ∗ is associative and distributes over + so (G,+,∗) satisfies all ring axioms with the exception of the commutativity of (G,+).
Now there is no conflict with the other answers here because the weird ring I’ve just described has no multiplicative identity. Thus, I have tried to insert the word “unital” = “possessing a two-sided identity” in various appropriate places in the above text in order to emphasize the assumption that an identity exists. Now if one does not assume the existence of an identity things can get complicated rather quickly. For instance, an arbitrary set M can be made into a non-unital monoid by choosing a point x∈M and imposing the law of composition (m,n)↦x for every pair (m,n)∈M×M. This is clearly an associative law of composition, but there is no identity and, perhaps more importantly, the canonical homomorphism m∈M↦lm∈MM as defined above has only one point in its image (the point map onto x).
The point is that if M does not contain an identity then m↦lm is not necessarily injective so one does not necessarily see an exact copy of M inside of MM – only a quotient. This means that the abstract definition of a (not necessarily unital) monoid can produce examples which are not canonically equivalent to the guiding example of SS and its submonoids, so in this case the definition is not “just language”, as I wrote above.
There is a sufficient condition for the injectivity of m↦lm which is more general than the existence of an identity element: if the right anti-representation x↦rx∈MM of M is defined as usual (rx(m)=mx) then m↦lm will be injective provided that there is at least one x∈M such that rx∈MM is an injective map, for then lm=ln implies
and therefore m=n by the injectivity of rx. I have a feeling that this is a manifestation of some more general phenomenon involving monomorphisms in the category of sets but I don’t know enough about category theory to discuss this in any detail (maybe @Martin Brandenburg would like to leave a comment). In particular, if M contains a two-sided identity 1 then r1∈MM is injective so m↦lm injective.
At the level of rings, this means that in an abstractly defined ring, not necessarily with identity, the presence of at least one x∈R such that rx∈End(R) is injective forces the canonical representation m↦lm to be injective and therefore to faithfully reproduce R inside of End(R).
Now alot has been said on this thread about how the distributive property in a ring forces the underlying group to be abelian and the computation presented by Bill Dubuque and drhab uses the existence of a ring identity to show this. In fact, this can be proved assuming only that the canonical representation is injective:
Proposition. If (R,+,∗) satisfies all the ring axioms except the commutativity of the group (R,+), then the canonical representation x↦[z↦lx(z)=x∗z] is a homomorphism of both products into RR which takes values in End(R) and if this homomorphism is injective then (R,+) is abelian.
Remark. Since the proposition requires two-sided distributivity of ∗ over +, the hypotheses are somewhat stronger than simply stating that (R,+,∗) is a near ring.
1.”x↦lx takes values in End(G)” uses distributivity from the left:
2.”x↦lx is a homomorphism ∗→∘” uses associativity of ∗:
3.”x↦lx is a homomorphism +→+” uses distributivity from the right:
4.(R,+) is abelian if x↦lx is injective:
Canceling the outer terms (which is valid since (R,+) is assumed to be a group), we have lx+y(z)=ly+x(z). This being true for all z, we conclude that x+y=y+x provided that x↦lx is injective. The proposition is proved.
What if we drop the assumption that the canonical representation is injective? Then we can produce examples of “rings with nonabelian underlying groups” as I did at the beginning of the edit – to reiterate: just take your favorite nonabelian group G written with + (which is atrocious, I know) and impose a second law of composition ∗ given by (x,y)↦x∗y=eG. It is easily verified that ∗ is associative and distributes over + so (G,+,∗) satisfies all ring axioms with the exception of the commutativity of (G,+). The proposition shows that the canonical representation cannot be injective and sure enough, it takes values only on a single point: the trivial endomorphism [x↦eG]∈End(G).
To sum things up I would say that
Yes, the ring axioms can be relaxed to produce “rings with nonabelian underlying groups”, however some nice property will have to be sacrificed: either
multiplication will not distribute from the left, which can give you a near-ring but then the image of the canonical representation will not lie in End(R), in general; or
the canonical representation will not be injective in which case you cannot adjoin an identity without severely disrupting the given algebraic structure (I assume that you will have to pass to some abelian quotient of the underlying group).
In either case (and especially in the second case) these objects will not interact well with Z, and in my estimation that is why they are mostly curiosities rather than the object of intense study.